Local patching of fonts

[Paul Flo Williams]

I've recently been looking at the prospect of locally patching font files
for Fedora, prompted by two things:

1. Petr Vobornik raised the issue of fonts having an fsType flag that
prevented embedding, even though they are licensed under terms which allow
distribution and modification.[1]

2. Bug 706559, "Font variants not used correctly"[2], refers to a font
family that contains more than just the R,B,I,BI styles and shows that
they are not well-treated by applications including LibreOffice. It
appears that this may be because the various weights and styles are not
well named, but we can't quite get to the bottom of this bug because the
family under discussion, Neo Sans Intel, is proprietary (i.e. not
packaged).

However, we can see bugs like 706559 in fonts that we have already
packaged. If you install chisholm-letterslaughing-fonts, you'll find that
you have three font files but only one is accessible in LibreOffice. Close
examination will show that all three have the Family name set to "Letters
Laughing", but the Subfamily (Style) names are "at Their Execution", "by
Quantized and Calibrated" and "Dissection and Destruction". You can see
why LO might be confused.[#]

In other cases, we have had to adopt rather awkward Fontconfig
configurations in order to rename fonts.

Now, we either fix font problems like these locally, or we push them back
upstream. We are supposed to do the latter, but we know how many fonts are
pushed out into the world as a one off. When font problems are raised
during review and the packager is asked to push the comments back
upstream, how often do we see little response and no new release?

Being practical, if we wish to fix problems locally, I'm aware of three
options:

1. FontTools/TTX, which can convert a font into XML and back again.
However, Petr reports that it currently crashes on Open Sans[3], and we
have no smart way of verifying that the "To XML, patch, compile to TTF"
cycle produces a font that performs at least as well as the original.

2. FontForge can be scripted to perform changes on fonts, but in this case
we compile a completely new font, which is subject to any number of quirks
introduced by local FontForge preferences, unstated in the script file,
and FF's well-known stability issues.

3. Fontconfig scripts, which most of us just hack from the templates
provided by Nicolas Mailhot because ... well, I certainly couldn't write
the DejaVu ones from scratch. There is the additional problem that not all
applications might use Fontconfig. (I might be misremembering but doesn't
Firefox do something special?)

The problem of embedding flags is now being addressed by the packaging of
a simple tool called "ttembed" which just touches OS/2.fsType and
checksums, nothing else.[4]

I was considering the construction of another, more general, tool that
would perform adjustments to other OS/2 or name table fields in-place[*],
adjusting checksums as necessary in order to keep the font consistent, but
specifically not decompiling the font or attempting to do glyph or GPOS
type changes.

The proposed method would be to write a very simple recipe for the font
changes, like this:

recipe.txt:

OS/2.fsType=0
  OS/2.panose=4,0,0,0,0,0,0,0,0,0
  name.1.0.0.1="Letters Laughing at Their Execution"
  name.1.0.0.2="Regular"
  name.1.0.0.4="Letters Laughing at Their Execution"
  name.3.1.1033.1="Letters Laughing at Their Execution"
  name.3.1.1033.2="Regular"
  name.3.1.1033.4="Letters Laughing at Their Execution"

and apply it with:

$ ttpatch recipe.txt LettersLaughing.ttf

With simple recipes like these, we could correct the entire Letters
Laughing family and any others that don't follow WWS naming. With local
testing for correct behaviour, we would even have a complete set of
corrections to suggest to upstream (with the expectation that they'd apply
them in their font editor, of course, rather than ttpatch.)

Note that the patch recipe contains deliberately almost-duplicated lines;
I'd suggest that it's better to repeat the name record lines for Mac and
Windows compatibility rather than try to build a cleverer ttpatch that
would try to keep the names consistent across platforms and fail in the
case of localisation differences, for instance.

I think that this would be a step forward from just packaging fonts and
accepting their quirks towards more careful curation. I'd like to hear of
any other problems with fonts in common applications that could be
addressed in this fashion, or any other thoughts?

[*] OK, "in-place" is stretching it when considering the name table, which
might grow or shrink as required, but I'm saying that this tool should
definitely not re-order tables or touch anything other than the tables
required to do the patch, which will be the touched tables, offset table
and the head.checkSumAdjustment field. This makes verifying the changes
much simpler.

[#] This issue came up during package review[5], and upstream refused to
rename the styles. I'm proposing to adjust the fonts so that their quirky
names remain unchanged, but redistributed amongst the name table fields so
that all variants are accessible. I doubt upstream really wanted 2 out of
3 designs to be unusable in our major office application.

[1] https://lists.fedoraproject.org/pipermail/devel/2013-November/192501.html
[2] https://bugzilla.redhat.com/show_bug.cgi?id=706559
[3] https://lists.fedoraproject.org/pipermail/devel/2013-November/192540.html
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1036754
[5] https://bugzilla.redhat.com/show_bug.cgi?id=491530

[Nicolas Mailhot]

Le Jeu 5 décembre 2013 13:29, Paul Flo Williams a écrit :
> I've recently been looking at the prospect of locally patching font files
> for Fedora, prompted by two things:
> Bug 706559, "Font variants not used correctly"[2], refers to a font
>
> family that contains more than just the R,B,I,BI styles and shows that
> they are not well-treated by applications including LibreOffice.

More than R,B,I,BI is fine and should be encouraged as long as the style
names conform to WWS. Some apps do not understand WWS, true but other apps
do not understand any style or fail on new font formats and we don't give
up on them. IMHO WWS has been the reasonable middle ground for some years.

DejaVu uses more than R,B,I,BI and should have shaken up WWS bugs in most
important apps by now. The fonts Adobe released also use more than
R,B,I,BI to put more pressure on app authors.
> Fontconfig scripts, which most of us just hack from the templates
>
> provided by Nicolas Mailhot because ... well, I certainly couldn't write
> the DejaVu ones from scratch.

I admit I spent quite a lot of time pestering Behdad on IRC to get advice :p
> There is the additional problem that not all
> applications might use Fontconfig. (I might be misremembering but doesn't
> Firefox do something special?)

Not interesting case, we want applications to use fontconfig, there are so
many things that break otherwise it's counter-productive to help
non-fontconfig users.

But of course fixing the font files is better than work-arounding them at
fontconfig level.
> The problem of embedding flags is now being addressed by the packaging of
> a simple tool called "ttembed" which just touches OS/2.fsType and
> checksums, nothing else.[4]

That was great!
> I was considering the construction of another, more general, tool that
> would perform adjustments to other OS/2 or name table fields in-place[*],
> adjusting checksums as necessary in order to keep the font consistent, but
> specifically not decompiling the font or attempting to do glyph or GPOS
> type changes.
>
> The proposed method would be to write a very simple recipe for the font
> changes, like this:

If you want to work on this that would be terrific. Ideally you'd need a
tool with two modes :

1. analysis mode: read a font file, try to guess correct WWS naming
(consistent with MS paper and existing metadata), print it. There is
already a crude name-guessing script in repo-font-audit (but it does not
check if the suggested weights & widths are consistent with the font
contents)

2. fix mode: given a font file, a family and WWS font name in input, fixes
all layers of metadata to be consistent with the proposed naming. Aborts
if naming is not WWS-compliant.

3. "automated" mode : string steps 1 and 2

That's about the most complex workflow you can propose to Fedora
packagers. Don't expect them to know the different opentype metadata
layers, just fix all the layers in one go (and add the missing ones).
Provide a way for translators to contribute translations of standard style
names, and inject all the translations the tools knows of at fix time.

That will harm interop in the case of very badly named fonts, because the
font in Fedora won't have the bugs people workaround in other systems, but
the only way to avoid this would be to add renaming heuristics at
fontconfig level, like Microsoft did (keeping the bugs in the fonts but
fixing the naming displayed in their apps). We hoped for those for years,
time to move on if you're ready to work on the problem.

It's useless to preserve Windows/Mac naming, depending on the OS the font
creator used it's almost certain that the one he didn't use has broken
naming. Better to have completely correct and consistent naming than
slightly-broken-in-some-cases fonts (at the oldest naming layer style
names will end up fully or in part in the font name which is ok as long as
it's done consistently)

You also need a tool to fix the copyright/licensing/distribution fields as
those are often wrong.

I think that in many cases upstreams will even adopt the result as I
suspect reluctance to fix the fonts is directly caused by atrocious
metadata edition UI in their editor of choice.

I'm really sad to be almost unavailable while such exciting changes happen
in Fedora fonts:(

Regards,

On 5 December 2013 17:59, Paul Flo Williams  wrote:
> FontTools/TTX, which can convert a font into XML and back again.
>
> However, Petr reports that it currently crashes on Open Sans[3], and we
> have no smart way of verifying that the "To XML, patch, compile to TTF"
> cycle produces a font that performs at least as well as the original.

I have done same in Liberation fonts 2.00.1 to fix Monospace bit. Works
quite well. [x]

x.
https://git.fedorahosted.org/cgit/liberation-fonts.git/plain/scripts/setisFi...
> The problem of embedding flags is now being addressed by the packaging of
> a simple tool called "ttembed" which just touches OS/2.fsType and
> checksums, nothing else.[4]

Thats great and very useful.
> The proposed method would be to write a very simple recipe for the font
> changes, like this:
>
> recipe.txt:
>
> OS/2.fsType=0
>   OS/2.panose=4,0,0,0,0,0,0,0,0,0
>   name.1.0.0.1="Letters Laughing at Their Execution"
>   name.1.0.0.2="Regular"
>   name.1.0.0.4="Letters Laughing at Their Execution"
>   name.3.1.1033.1="Letters Laughing at Their Execution"
>   name.3.1.1033.2="Regular"
>   name.3.1.1033.4="Letters Laughing at Their Execution"
>
> and apply it with:
>
> $ ttpatch recipe.txt LettersLaughing.ttf

Yes, in long term increasing the scope can help lot.

In Fedora we do recommend to build fonts from sources as patching source
.sfd is an efficient method. But we get number of upstream releases only
with source so dont have option. Having script to fix ttf metadata is
definitely going to help to number of projects.

Regards,
Pravin Satpute

[Dave Crossland]

On 6 December 2013 00:01, [e-mail]  wrote:
> On 5 December 2013 17:59, Paul Flo Williams  wrote:
> > FontTools/TTX, which can convert a font into XML and back again.
> >
> > However, Petr reports that it currently crashes on Open Sans[3], and we
> > have no smart way of verifying that the "To XML, patch, compile to TTF"
> > cycle produces a font that performs at least as well as the original.

Why is this? You can ttx both files and diff them :)

Crash could be because of the old version. Behdad is now actively
maintaining a fork of the project -
https://github.com/behdad/fonttools/ - which may not crash.
> I have done same in Liberation fonts 2.00.1 to fix Monospace bit. Works
> quite well. [x]

:)
> > The problem of embedding flags is now being addressed by the packaging of
> > a simple tool called "tt
>
> In Fedora we do recommend to build fonts from sources as patching source
> .sfd is an efficient method. But we get number of upstream releases only
> with source so dont have option. Having script to fix ttf metadata is
> definitely going to help to number of projects.

In https://github.com/xen/fontbakery/ the developer is landing a
commit in the next few days to build fonts from TTX source in addition
to the current UFO workflow; and another developer, Franke Trampe, is
working a lot on FontForge's UFO support this month.

[Behdad Esfahbod]

On 13-12-05 07:29 AM, Paul Flo Williams wrote:
> FontTools/TTX, which can convert a font into XML and back again.
>
> However, Petr reports that it currently crashes on Open Sans[3],

I can't reproduce this.  Details?

[T.C. Hollingsworth]

On Thu, Dec 5, 2013 at 5:29 AM, Paul Flo Williams  wrote:
> FontTools/TTX, which can convert a font into XML and back again.
>
> However, Petr reports that it currently crashes on Open Sans[3], and we
> have no smart way of verifying that the "To XML, patch, compile to TTF"
> cycle produces a font that performs at least as well as the original.

As mentioned downthread, we really ought to switch to Behdad's fork of
fonttools in Fedora.  It would solve a lot of issues we're having.
> I was considering the construction of another, more general, tool that
> would perform adjustments to other OS/2 or name table fields in-place[*],
> adjusting checksums as necessary in order to keep the font consistent, but
> specifically not decompiling the font or attempting to do glyph or GPOS
> type changes.
>
> The proposed method would be to write a very simple recipe for the font
> changes, like this:

I wrote a little utility called ttname for updating the name table to
address concerns about licensing metadata not matching for webfonts.
(Just `yum install ttname` to use it.)  It uses ttx under the hood,
but it avoids serializing anything but the name table to XML to avoid
potential issues messing up the other tables.  (Not that there should
be any issues according to Behdad, who definitely would know.  :-)

With it, this:
> name.1.0.0.1="Letters Laughing at Their Execution"
>   name.1.0.0.2="Regular"
>   name.1.0.0.4="Letters Laughing at Their Execution"

is just:

ttname -p1 -e0 -l0 \
  --name1="Letters Laughing at Their Execution" \
  --name2="Regular" \
  --name4="Letters Laughing at Their Execution" \
  font.otf

It also has an -a switch to perform the same changes on all
platform/language/encoding combinations instead of having to
inidvidually specify them, and some named arguments for those who
don't know the numbers of the top of their head, so more succinctly:

ttname -a \
  --family="Letters Laughing at Their Execution" \
  --subfamily="Regular" \
  --name="Letters Laughing at Their Execution" \
  font.ttf

It doesn't currently support modifying the OS/2 table but I could add
support for that fairly easily.  (Of course, then the name wouldn't be
very accurate, but oh well.  ;-)
> Note that the patch recipe contains deliberately almost-duplicated lines;
> I'd suggest that it's better to repeat the name record lines for Mac and
> Windows compatibility rather than try to build a cleverer ttpatch that
> would try to keep the names consistent across platforms and fail in the
> case of localisation differences, for instance.

99% of the time there's only one language in the name table, so I
figured the -a switch would be useful.  You can still specify the
individual platform/encoding/lang triplets for the edge cases.

-T.C.