Request for review and advice on wqy-bitmap-fonts fontconfig settings

[Qianqian Fang]

hi

I am quite new as a Fedora package maintainer. I submitted
a few Chinese font packages to Fedora, and luckily, Jens Petersen
has been extremely patient and guided me through
package submission processes. I am glad that these fonts are
now serving the Chinese Fedora users and we got quite good
feedbacks from them.

Today, I would like to ask for your help to create a robust
fontconfig file for wqy-bitmap-fonts. This font was installed
by default in Fedora 8 for Chinese users. It recently caused
some problems displaying Latins for non-Chinese users.
Please see bug #381311 for more details.
https://bugzilla.redhat.com/show_bug.cgi?id=381311

I will describe my goals, the achieved results and the
problems.

===================================
1. Motivations and goals

As an active developer and Linux user for couple of years,
I strongly feel that there is a consensus among most Chinese users
(both Mainland and Taiwan) for font rendering. These consensus
can be summarized as the followings:

1. given the fact that most Chinese vector font rendering are
   quite blurry on nearly all Linux distributions, plus the fact that 
there is
   no free Chinese fonts with high quality hinting, hand-tuned
   bitmap glyphs are preferred for on screen display of Chinese characters
2. Latin glyphs has low stroke density and autohinting is becoming
   sufficient, using vector glyphs of these non-CJK characters is preferred
3. ideally, the requirement #1 should be locale independent (maybe 
exclude ko/ja
   users, if it conflicts with their consensus); requirement #2 is virtually
   true for most modern Linux desktops nowadays (for non-CJK locales)

The following two pictures may shine some light on what
a normal Chinese user considers as a "good" font rendering:

* under non-Chinese locales (use en_US as an example)
[can not be achieved for now, used post-processing]
http://wenq.org/gallery/albums/userpics/10002/confopt_preferred_rendering_en...

* under Chinese locales (use zh_TW as an example)
[achieved on F8 with wqy-bitmap-fonts 0.9.9-1]
http://wenq.org/gallery/albums/userpics/10002/confopt_preferred_rendering_zh...

the main features include:
A. when rendering Hanzi for generic aliases (i.e. sans/serif/mono):
A.1: if font sizes are common on screen, such as 8pt-12pt, use bitmap 
Chinese font
A.2: for sizes above or below, use the first preferred vector Chinese font
B. when rendering non-Hanzi (or non-CJK) glyphs, use the preferred fonts 
determined by fontconfig
C. when a specific Chinese font was chosen, exclusively use this font 
for all covered characters

===================================
2. Default rendering of Hanzi on F8 without wqy-bitmap-fonts

Without installing wqy-bitmap-fonts, the screenshots of F8 is shown below:

* under non-Chinese locales (use en_US as an example)
http://wenq.org/gallery/albums/userpics/10002/confopt_F8_no_wqy-bitmap_en-us...

unsatisfactories:
1) no bitmap fonts were used for screen-sized Hanzi
2) garbled text with a mixture of Japanese and Chinese fonts (the headings)
3) Hanzi glyphs are blurry, getting worse for large blocks of text

* under Chinese locales (use zh_TW as an example)
http://wenq.org/gallery/albums/userpics/10002/confopt_F8_no_wqy-bitmap_zh.pn...

unsatisfactories:
1) generally looks OK if Uming is installed
2) for mono, the Latins in Uming were used, rather than Dejavu/Vera
3) for serif, no bitmap glyphs because UKai has no embedded bitmaps

===================================
3. Hanzi rendering on F8 with 85-wqy-bitmapsong.conf

The devel. of wqy-bitmap-fonts started from expanding the embedded 
bitmap fonts
in Uming (both originated from firefly-bitmap-font) 3 years ago. After
3 years development at wenq.org, our project website, we have completed
all bitmap glyphs for CJK Basic (U4E00-U9FA5) and CJK Extension A
(U3400-U4DB5), covering 27,484 characters at 4 point sizes, nearly
twice of the Uming's embed bitmaps (only ~15,000 characters). In addition,
80% of the old firefly(uming) glyphs were fine-tuned. The improvements are
quite significant, making the font a popular choice among Chinese users.
The following sample shows the difference between uming and 
wqy-bitmap-fonts:

http://wenq.org/gallery/albums/userpics/10002/confopt_wqy-bitmap_vs_uming.pn...

In package wqy-bitmap-fonts, we provided a default fontconfig file,
85-wqy-bitmapsong.conf (see attachment). Using this file, we raised
the priority of wqy-bitmap-fonts for rendering Hanzi, while trying to 
keep it
lower than the default sans/serif/mono Latin fonts. With this
file, Chinese users are able to get the preferred rendering as
shown above, i.e.
http://wenq.org/gallery/albums/userpics/10002/confopt_preferred_rendering_zh...

however, for English locales, it did not seem to improve the situation, nor
make it worse (at least with my copy)
http://wenq.org/gallery/albums/userpics/10002/en_US_with_wqy-bitmap-fonts.pn...

I do noticed that this file has side effects, the bug thread (#381311) 
as an example,
however, so far I have not yet been able to identify the exact cause. 
These reports
are rather random and mostly non-repeatable. Debugging fontconfig outputs
constantly gives me confusing results. My guess is that we might use some
fragile fontconfig syntax and are not consistently executed over 
different systems.

===================================
4. Questions

That's all I want to learn from you: do you see a robust implementation
of fontconfig font selection mechanism to achieve my goals above?
if yes, how? if no, to whom should I file bug reports to?

I apologize for the long email, I wish you read to this line before 
giving up.

thank you so much and looking forward to hearing back from you.

Qianqian

[Nicolas Mailhot]

Le vendredi 23 novembre 2007 à 23:25 -0500, Qianqian Fang a écrit :

Hi Qianqian,
> Questions
>
> That's all I want to learn from you: do you see a robust implementation
> of fontconfig font selection mechanism to achieve my goals above?
> if yes, how? if no, to whom should I file bug reports to?

I'm not sure if I've assimilated all your goals, but here are some
comments on your fontconfig file:
> <?xml version="1.0"?>
>
> <!DOCTYPE fontconfig SYSTEM "fonts.dtd">

You probably want ../fonts.dtd there but that's general XML fontconfig
brokenness most everyone is guilty of.
> <match target="pattern">

I suppose this is protected by the selectfont pattern before but I'd
avoid playing any complex substitution games when the name of an
existing on-system font is given, and only substitute either fonts not
available at all or synthetic fonts like sans-serif and friends.

Also, please use a simple match with a test for family inside instead of
this pattern indirection, as documented on:
http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig
>     <test equal="any" compare="eq" name="lang">
>             <string>en</string>
>             <string>en-us</string>

You have no business touching non CJK locales, please remove this. If
for some reason an app does not tag text with the correct locale,
complain to its authors.

http://fedoraproject.org/wiki/SIGs/Fonts/Dev/LanguageAwarenessProblem

These two lines are probably responsible for most of your problem
reports.
>             <string>zh-cn</string>
>             <string>zh-tw</string>
>             <string>zh-hk</string>
>             <string>zh-sg</string>
>     </test>
>     <test compare="more_eq" name="pixelsize">
>             <double>11</double>
>     </test>
>     <test compare="less_eq" name="pixelsize">
>             <double>16</double>
>     </test>

Are you sure of the 11-16 pixel range?
>     <edit name="family" mode="prepend_first">
>             <string>WenQuanYi Bitmap Song</string>
>     </edit>
> </match>
> <match target="pattern">

[…]
>     <test compare="more_eq" name="size">
>             <double>8</double>
>     </test>
>     <test compare="less_eq" name="size">
>             <double>12</double>
>     </test>

Point sizes have no meaning for bitmap fonts, unless you assume a fixed
point/pixel ratio, which is not hardware reality. Dump this match block
it's actively evil.
>     <edit name="family" mode="prepend_first">
>             <string>WenQuanYi Bitmap Song</string>
>     </edit>
>
>    </match>
> <match target="pattern">
>     <test equal="any" compare="eq" name="family">
>             <string>WenQuanYi Bitmap Song</string>
>     </test>
>     <test equal="any" compare="eq" name="family">
>             <string>monospace</string>
>     </test>
>     <edit name="family" mode="prepend">
>             <string>Bitstream Vera Sans Mono</string>
>     </edit>
>     <edit name="family" mode="prepend" binding="strong">
>             <string>DejaVu LGC Sans Mono</string>
>     </edit>
> </match>

If you didn't touch non-CJK locales before you wouldn't need this
bandaid. Also you're assuming some other fonts are present on system
which may not be the case. Our default latin font list is dynamic and
changes from release to release, and depending on what font packages
users actually install.

If you really want Vera or DejaVu to override WenQuanYi Bitmap Song
contents, the override should be added to the fontconfig file those
fonts ship.

Wouldn't a simple 
http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig#local-specific...
rule with additional pixel range restriction be sufficient for your
needs?

Regards,

[Qianqian Fang]

hi Nicolas

the syntax described in
Fonts/Packaging/Fontconfighttp://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfigseems
not sufficient
to accomplish the desired substitution rules.

First, we want to set wqy-bitmap-fonts as the default font for displaying
Chinese characters (for example U4E00-U9FA5, and U3400-U4D86 and some
punctuations), for all aliases (sans/serif/mono), and all locales (maybe
except ja/ko), only at 10px to 16px. This can not be done with
<alias><prefer></prefer></alias> syntax (I just tested, it used the Latin
part in wqy-bitmap-fonts in mono, and can not use Uming/ukai for above/below
10-16px, both of these are bad).

Second, I want the system preferred Latin fonts to display the non-CJK part.
The provided syntax, in my own opinion, can not do this either.

You mention "no business touching non CJK locales," I don't fully agree. The
default fontconfig setup for rendering Chinese webpages under non-zh locales
is terrible. You can test this using en or other non CJK locales, by
browsing http://wenq.org , you will see a mixed text rendered by Japanese
fonts, uming, ukai and some Japanese/Korean bitmap X core fonts. In my
opinion, that is non-usable. To tell the truth, en_US is my default locale
for both office and at home. I don't want my desktop looks non-legible by my
colleagues. However, I do want it to process Chinese properly. Most my
Chinese friends working in the US have the same set up for the same reason.
The default en (or non CJK) Chinese rendering does need to be improved (it
might be more appropriate to handle this with Fedora font committee or some
sort, but improving the situation by installing an additional font package
may not a bad solution either: those who don't like it just uninstall this
font, and everything will back to the default way).

Last, the link that you mentioned does look promising:
http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig#local-specific...
However, my question is, how can I make sure that the preferred Latin fonts
(not necessarily Vera/Dejavu, could be something that user chose as in
Andreas's case in bug #381311) overrides the Latin part of wqy?

Thank you and I would be glad to hear further instructions.

Qianqian

On Nov 24, 2007 9:21 AM, Nicolas Mailhot  wrote:
> Le vendredi 23 novembre 2007 à 23:25 -0500, Qianqian Fang a écrit :
>
> Hi Qianqian,
> > Questions
> >
> > That's all I want to learn from you: do you see a robust implementation
> > of fontconfig font selection mechanism to achieve my goals above?
> > if yes, how? if no, to whom should I file bug reports to?
>
> I'm not sure if I've assimilated all your goals, but here are some
> comments on your fontconfig file:
> > <?xml version="1.0"?>
> >
> > <!DOCTYPE fontconfig SYSTEM "fonts.dtd">
>
> You probably want ../fonts.dtd there but that's general XML fontconfig
> brokenness most everyone is guilty of.
> > <match target="pattern">
>
> I suppose this is protected by the selectfont pattern before but I'd
> avoid playing any complex substitution games when the name of an
> existing on-system font is given, and only substitute either fonts not
> available at all or synthetic fonts like sans-serif and friends.
>
> Also, please use a simple match with a test for family inside instead of
> this pattern indirection, as documented on:
> http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig
> >     <test equal="any" compare="eq" name="lang">
> >             <string>en</string>
> >             <string>en-us</string>
>
> You have no business touching non CJK locales, please remove this. If
> for some reason an app does not tag text with the correct locale,
> complain to its authors.
>
> http://fedoraproject.org/wiki/SIGs/Fonts/Dev/LanguageAwarenessProblem
>
> These two lines are probably responsible for most of your problem
> reports.
> >             <string>zh-cn</string>
> >             <string>zh-tw</string>
> >             <string>zh-hk</string>
> >             <string>zh-sg</string>
> >     </test>
> >     <test compare="more_eq" name="pixelsize">
> >             <double>11</double>
> >     </test>
> >     <test compare="less_eq" name="pixelsize">
> >             <double>16</double>
> >     </test>
>
> Are you sure of the 11-16 pixel range?
> >     <edit name="family" mode="prepend_first">
> >             <string>WenQuanYi Bitmap Song</string>
> >     </edit>
> > </match>
> > <match target="pattern">
>
> […]
> >     <test compare="more_eq" name="size">
> >             <double>8</double>
> >     </test>
> >     <test compare="less_eq" name="size">
> >             <double>12</double>
> >     </test>
>
> Point sizes have no meaning for bitmap fonts, unless you assume a fixed
> point/pixel ratio, which is not hardware reality. Dump this match block
> it's actively evil.
> >     <edit name="family" mode="prepend_first">
> >             <string>WenQuanYi Bitmap Song</string>
> >     </edit>
> >
> >    </match>
> > <match target="pattern">
> >     <test equal="any" compare="eq" name="family">
> >             <string>WenQuanYi Bitmap Song</string>
> >     </test>
> >     <test equal="any" compare="eq" name="family">
> >             <string>monospace</string>
> >     </test>
> >     <edit name="family" mode="prepend">
> >             <string>Bitstream Vera Sans Mono</string>
> >     </edit>
> >     <edit name="family" mode="prepend" binding="strong">
> >             <string>DejaVu LGC Sans Mono</string>
> >     </edit>
> > </match>
>
> If you didn't touch non-CJK locales before you wouldn't need this
> bandaid. Also you're assuming some other fonts are present on system
> which may not be the case. Our default latin font list is dynamic and
> changes from release to release, and depending on what font packages
> users actually install.
>
> If you really want Vera or DejaVu to override WenQuanYi Bitmap Song
> contents, the override should be added to the fontconfig file those
> fonts ship.
>
> Wouldn't a simple
>
> http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig#local-specific...
> rule with additional pixel range restriction be sufficient for your
> needs?
>
> Regards,
>
> --
> Nicolas Mailhot

[Qianqian Fang]

hi

I am making some progress using the syntax in 
SIGs/Fonts/Packaging/Fontconfig 
http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig#local-specific-default-font-overrides, 
however,
there are still problems. I hope either of you can help me find out 
what's wrong.

the new fontconfig file is attached. I used <alias><prefer> environments,
and renamed the file from 85 to 61 (it does not work at 85), inserted 
between
60-latin and 64-nonlatin-fedora.

Now, for en locale, almost perfect, except that the last two
blocks did not seem to work (i.e. replacing WenQuanYi Bitmap Song by
Uming when pixelsize>16 and <11).  You can browse
http://wqy.sourceforge.net/cgi-bin/eindex.cgi?WQYTest
to test.

For zh locales, in addition to the above problem, under monospace 
(gnome-terminal),
the Latin glyphs were rendered by the wqy bitmap glyphs. I expect
those are rendered by Dejavu Sans Mono or Bitstream Sans Mono.

in either case, the output of
FC_DEBUG=4 fc-match "monospace:lang=zh-cn:pixelsize=10px"
always put wqy as the first one, this is not consistent with what
Firefox actually displayed under en locales.

Can any of you help me to find out how to get around these problems?

thank you

Qianqian

Qianqian Fang wrote:
> hi Nicolas
>
> the syntax described in Fonts/Packaging/Fontconfig 
> http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig seems 
> not sufficient
> to accomplish the desired substitution rules.
>
> First, we want to set wqy-bitmap-fonts as the default font for 
> displaying Chinese characters (for example U4E00-U9FA5, and 
> U3400-U4D86 and some punctuations), for all aliases (sans/serif/mono), 
> and all locales (maybe except ja/ko), only at 10px to 16px. This can 
> not be done with <alias><prefer></prefer></alias> syntax (I just 
> tested, it used the Latin part in wqy-bitmap-fonts in mono, and can 
> not use Uming/ukai for above/below 10-16px, both of these are bad).
>
> Second, I want the system preferred Latin fonts to display the non-CJK 
> part. The provided syntax, in my own opinion, can not do this either.
>
> You mention "no business touching non CJK locales," I don't fully 
> agree. The default fontconfig setup for rendering Chinese webpages 
> under non-zh locales is terrible. You can test this using en or other 
> non CJK locales, by browsing http://wenq.org , you will see a mixed 
> text rendered by Japanese fonts, uming, ukai and some Japanese/Korean 
> bitmap X core fonts. In my opinion, that is non-usable. To tell the 
> truth, en_US is my default locale for both office and at home. I don't 
> want my desktop looks non-legible by my colleagues. However, I do want 
> it to process Chinese properly. Most my Chinese friends working in the 
> US have the same set up for the same reason. The default en (or non 
> CJK) Chinese rendering does need to be improved (it might be more 
> appropriate to handle this with Fedora font committee or some sort, 
> but improving the situation by installing an additional font package 
> may not a bad solution either: those who don't like it just uninstall 
> this font, and everything will back to the default way).
>
> Last, the link that you mentioned does look promising:
> http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig#local-specific... 
> http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig#local-specific-default-font-overrides
> However, my question is, how can I make sure that the preferred Latin 
> fonts (not necessarily Vera/Dejavu, could be something that user chose 
> as in Andreas's case in bug #381311) overrides the Latin part of wqy?
>
> Thank you and I would be glad to hear further instructions.
>
> Qianqian
>
> On Nov 24, 2007 9:21 AM, Nicolas Mailhot <[e-mail] 
> > wrote:
> Le vendredi 23 novembre 2007 à 23:25 -0500, Qianqian Fang a écrit :
>
> Hi Qianqian,
>
> > 4. Questions
> >
> > That's all I want to learn from you: do you see a robust
> implementation
> > of fontconfig font selection mechanism to achieve my goals above?
> > if yes, how? if no, to whom should I file bug reports to?
>
> I'm not sure if I've assimilated all your goals, but here are some
> comments on your fontconfig file:
>
> > <?xml version=" 1.0"?>
> > <!DOCTYPE fontconfig SYSTEM "fonts.dtd">
>
> You probably want ../fonts.dtd there but that's general XML fontconfig
> brokenness most everyone is guilty of.
>
> >     <match target="pattern">
>
> I suppose this is protected by the selectfont pattern before but I'd
> avoid playing any complex substitution games when the name of an
> existing on-system font is given, and only substitute either fonts
> not
> available at all or synthetic fonts like sans-serif and friends.
>
> Also, please use a simple match with a test for family inside
> instead of
> this pattern indirection, as documented on:
> http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig
>
> >         <test equal="any" compare="eq" name="lang">
>
> >                 <string>en</string>
> >                 <string>en-us</string>
>
> You have no business touching non CJK locales, please remove this. If
> for some reason an app does not tag text with the correct locale,
> complain to its authors.
>
> http://fedoraproject.org/wiki/SIGs/Fonts/Dev/LanguageAwarenessProblem
>
> These two lines are probably responsible for most of your problem
> reports.
>
> >                 <string>zh-cn</string>
> >                 <string>zh-tw</string>
> >                 <string>zh-hk</string>
> >                 <string>zh-sg</string>
> >         </test>
>
> >         <test compare="more_eq" name="pixelsize">
> >                 <double>11</double>
> >         </test>
> >         <test compare="less_eq" name="pixelsize">
> >                 <double>16</double>
> >         </test>
>
> Are you sure of the 11-16 pixel range?
>
> >         <edit name="family" mode="prepend_first">
> >                 <string>WenQuanYi Bitmap Song</string>
> >         </edit>
> >     </match>
>
> >     <match target="pattern">
>
> […]
>
> >         <test compare="more_eq" name="size">
> >                 <double>8</double>
> >         </test>
> >         <test compare="less_eq" name="size">
> >                 <double>12</double>
> >         </test>
>
> Point sizes have no meaning for bitmap fonts, unless you assume a
> fixed
> point/pixel ratio, which is not hardware reality. Dump this match
> block
> it's actively evil.
>
> >         <edit name="family" mode="prepend_first">
> >                 <string>WenQuanYi Bitmap Song</string>
> >         </edit>
> >    </match>
>
>
> >     <match target="pattern">
> >         <test equal="any" compare="eq" name="family">
> >                 <string>WenQuanYi Bitmap Song</string>
> >         </test>
> >         <test equal="any" compare="eq" name="family">
> >                 <string>monospace</string>
> >         </test>
> >         <edit name="family" mode="prepend">
> >                 <string>Bitstream Vera Sans Mono</string>
> >         </edit>
> >         <edit name="family" mode="prepend" binding="strong">
> >                 <string>DejaVu LGC Sans Mono</string>
> >         </edit>
> >     </match>
>
> If you didn't touch non-CJK locales before you wouldn't need this
> bandaid. Also you're assuming some other fonts are present on system
> which may not be the case. Our default latin font list is dynamic and
> changes from release to release, and depending on what font packages
> users actually install.
>
> If you really want Vera or DejaVu to override WenQuanYi Bitmap Song
> contents, the override should be added to the fontconfig file those
> fonts ship.
>
> Wouldn't a simple
> http://fedoraproject.org/wiki/SIGs/Fonts/Packaging/Fontconfig#local-specific-default-font-overrides
> rule with additional pixel range restriction be sufficient for your
> needs?
>
>
> Regards,
>
> --
> Nicolas Mailhot