[master 1/6] Initial framework and tests

[dashea]

From: David Shea

The initial set of tests is:
  - translated strings come back with the same markup
  - translated .po files don't just completely bomb out
  - translations are > 10% translated (warning only, since no anonymous
    pull support makes this one awkward for running in jenkins)
  - translatable strings don't contain unnecessary markup
  - single-character translatable strings include a comment explaining
    what in the heck

There is also a xgettext wrapper script in translatable that makes
xgettext warnings louder. This one is not run with the rest of the tests
since xgettext needs the original source file for these instead of just
the POT file.
---
 .gitignore                      |  2 +
 README.rst                      | 36 +++++++++++++++
 translatable/__init__.py        | 85 +++++++++++++++++++++++++++++++++++
 translatable/__main__.py        | 33 ++++++++++++++
 translatable/test_comment.py    | 26 +++++++++++
 translatable/test_markup.py     | 38 ++++++++++++++++
 translatable/xgettext_werror.sh | 46 +++++++++++++++++++
 translated/__init__.py          | 99 +++++++++++++++++++++++++++++++++++++++++
 translated/__main__.py          | 33 ++++++++++++++
 translated/test_markup.py       | 44 ++++++++++++++++++
 translated/test_percentage.py   | 42 +++++++++++++++++
 translated/test_usability.py    | 28 ++++++++++++
 12 files changed, 512 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 README.rst
 create mode 100644 translatable/__init__.py
 create mode 100644 translatable/__main__.py
 create mode 100644 translatable/test_comment.py
 create mode 100644 translatable/test_markup.py
 create mode 100755 translatable/xgettext_werror.sh
 create mode 100644 translated/__init__.py
 create mode 100644 translated/__main__.py
 create mode 100644 translated/test_markup.py
 create mode 100644 translated/test_percentage.py
 create mode 100644 translated/test_usability.py

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..8d35cb3
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,2 @@
+__pycache__
+*.pyc
diff --git a/README.rst b/README.rst
new file mode 100644
index 0000000..71df68b
--- /dev/null
+++ b/README.rst
@@ -0,0 +1,36 @@
+translation-canary
+-------------
+
+Translations can crash your program. Creating software for a wide audience
+means sending your strings away for translation, and giving up control of your
+strings means that strings with extralinguistic content can come back broken.
+No one is likely to even realize it until someone fires up your program in
+Hungarian and it crashes because Gtk bombed out on some busted markup, and the
+Hungarian speaker is sad, and you are sad, and everything is just the absolute
+worst.
+
+This is the canary in the translation coalmine.
+
+There are two parts to this project:
+
+translatable:
+  This contains checks on the strings to be submitted for translation. This
+  ensures that the content of the original strings marked for translation are
+  suitable for translation. These tests are run on the POT file before
+  uploading the POT or the updated PO files to the translators.
+
+translated:
+  This contains checks on the strings returned from the translators. This
+  ensures that the content of the translated strings won't break anything.
+  These tests are run on the source distribution before creating a release.
+
+Both translatable and translated are run by running the module
+(e.g., `python3 -m translatable`) with the input file(s) as the argument.
+
+In addition to the python modules, the translatable directory contains
+xgettext_werror.sh, a wrapper for xgettext that treats warnings as errors.
+xgettext will print warnings as it extracts translatable strings from source
+files, and these warnings should be addressed instead of silently ignored as
+they scroll by in the build output. To use the script in a package that uses
+the gettext template files from autopoint or gettextize, set
+XGETTEXT=/path/to/xgettext_werror.sh in Makevars.
diff --git a/translatable/__init__.py b/translatable/__init__.py
new file mode 100644
index 0000000..9531aa2
--- /dev/null
+++ b/translatable/__init__.py
@@ -0,0 +1,85 @@
+# Framework for testing translatable strings
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+"""
+Framework for running tests against translatable strings.
+
+Tests are loaded from modules in this directory. A test is any callable object
+within the module with a name that starts with 'test_'.
+
+Each test is called with a POEntry object as an argument. A test passes if it
+returns without raising an exception.
+"""
+
+try:
+    import polib
+except ImportError:
+    print("You need to install the python-polib package to read translations")
+    raise
+
+# Gather tests from this directory
+import pkgutil
+_tests = []
+for finder, mod_name, _ispkg in pkgutil.iter_modules(__path__):
+    # Skip __main__
+    if mod_name == "__main__":
+        continue
+
+    # Load the module
+    module = finder.find_module(mod_name).load_module()
+
+    # Look for attributes that start with 'test_' and add them to the test list
+    for attrname, attr in module.__dict__.items():
+        if attrname.startswith('test_') and callable(attr):
+            _tests.append(attr)
+
+def testString(poentry):
+    """Run all tests against the given translatable string.
+
+       :param polib.POEntry poentry: The PO file entry to test
+       :returns: whether the tests succeeded or not
+       :rtype: bool
+    """
+    success = True
+    for test in _tests:
+        try:
+            test(poentry)
+        except Exception as e:
+            success = False
+            print("%s failed on %s: %s" % (test.__name__, poentry.msgid, str(e)))
+    
+    return success
+
+def testPOT(potfile):
+    """Run all tests against all entries in a POT file.
+
+       :param str potfile: The name of a .pot file to test
+       :return: whether the checks succeeded or not
+       :rtype: bool
+    """
+    success = True
+
+    parsed_pot = polib.pofile(potfile)
+
+    for entry in parsed_pot:
+        if not testString(entry):
+            success = False
+
+    return success
diff --git a/translatable/__main__.py b/translatable/__main__.py
new file mode 100644
index 0000000..905cc70
--- /dev/null
+++ b/translatable/__main__.py
@@ -0,0 +1,33 @@
+# Entry point for testing translatable strings
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+import sys
+from . import testPOT
+
+if len(sys.argv) < 2:
+    print("Usage: translatable <POTfile>")
+    sys.exit(1)
+
+status = 0
+for potfile in sys.argv[1:]:
+    if not testPOT(potfile):
+        status = 1
+
+sys.exit(status)
diff --git a/translatable/test_comment.py b/translatable/test_comment.py
new file mode 100644
index 0000000..51410fb
--- /dev/null
+++ b/translatable/test_comment.py
@@ -0,0 +1,26 @@
+# Check that a string that needs a comment has one
+# # Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+def test_comment(poentry):
+    # Single-character translatable strings (e.g., the 'c' of press c to
+    # continue) need some additional context in order to make sense. Make
+    # sure that they have it.
+
+    if len(poentry.msgid) == 1 and not poentry.comment:
+        raise AssertionError("Single-character string missing a comment.")
diff --git a/translatable/test_markup.py b/translatable/test_markup.py
new file mode 100644
index 0000000..1faa6d6
--- /dev/null
+++ b/translatable/test_markup.py
@@ -0,0 +1,38 @@
+# Check that a string does not contain unnecessary Pango markup
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+from pocketlint.pangocheck import is_markup, markup_necessary
+import xml.etree.ElementTree as ET
+
+import gi
+gi.require_version("GLib", "2.0")
+from gi.repository import GLib
+
+def test_markup(poentry):
+    # Unnecessary markup is markup applied to an entire string, such as
+    # _("<b>Bold Text</b>"). This could be instead be translated as
+    # "<b>%s</b>" % _("Bold Text"), and then the translator doesn't have to see
+    # the markup at all.
+
+    if is_markup(poentry.msgid):
+        # Wrap the string in <markup> nodes, parse it, test it
+        tree = ET.fromstring("<markup>%s</markup>" % GLib.markup_escape_text(poentry.msgid))
+        if not markup_necessary(tree):
+            raise AssertionError("Unnecessary markup")
diff --git a/translatable/xgettext_werror.sh b/translatable/xgettext_werror.sh
new file mode 100755
index 0000000..36eefec
--- /dev/null
+++ b/translatable/xgettext_werror.sh
@@ -0,0 +1,46 @@
+#!/bin/sh -e
+#
+# xgettext_werror.sh: Run xgettext and actually do something with the warnings
+#
+# xgettext prints out warnings for certain problems in translatable strings,
+# such as format strings that cannot be translated due to position-based
+# parameters. These warnings generally indicate something that needs to be
+# addressed before the strings can be submitted for translation. This script
+# exits with a status of 1 so that the warnings are not ignored as they scroll
+# by in pages of build output.
+#
+# This script should be used in place of xgettext when rebuilding the .pot file,
+# e.g. by setting XGETTEXT in po/Makevars.
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+returncode=0
+
+# Collect the output from xgettext. If xgettext fails, treat that as a failure
+# Make sure that "warning:" doesn't get translated
+xgettext_output="$(LC_MESSAGES=C xgettext "$@" 2>&1)" || returncode=$?
+
+# Look for warnings
+if echo "$xgettext_output" | fgrep -q "warning: "; then
+    returncode=1
+fi
+
+# Print the output and return
+echo "$xgettext_output"
+exit $returncode
diff --git a/translated/__init__.py b/translated/__init__.py
new file mode 100644
index 0000000..af0af73
--- /dev/null
+++ b/translated/__init__.py
@@ -0,0 +1,99 @@
+# Framework for testing translations
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+"""
+Framework for running tests against translations.
+
+Tests are loaded from modules in this directory. A test is any callable object
+within the module with a name that starts with 'test_'.
+
+Each test is called with the name of .mo file to test as an argument. A test
+passes if it returns without raising an exception.
+"""
+
+import sys, tempfile, shutil, os, warnings
+
+_tests = []
+
+# Gather tests from this directory
+import pkgutil
+for finder, mod_name, _ispkg in pkgutil.iter_modules(__path__):
+    # Skip __main__
+    if mod_name == "__main__":
+        continue
+
+    # Load the module
+    module = finder.find_module(mod_name).load_module()
+
+    # Look for attributes that start with 'test_' and add them to the test list
+    for attrname, attr in module.__dict__.items():
+        if attrname.startswith('test_') and callable(attr):
+            _tests.append(attr)
+
+def testFile(mofile, prefix=None):
+    """Run all registered tests against the given .mo file.
+
+       :param str mofile: The .mo file name to check
+       :param str prefix: An optional directory prefix to strip from error messages
+       :return: whether the checks succeeded or not
+       :rtype: bool
+    """
+    success = True
+    for test in _tests:
+        # Don't print the tmpdir path in error messages
+        if prefix is not None and mofile.startswith(prefix):
+            moerror = mofile[len(prefix):]
+        else:
+            moerror = mofile
+
+        try:
+            with warnings.catch_warnings(record=True) as w:
+                test(mofile)
+
+                # Print any warnings collected
+                for warn in w:
+                    print("%s warned on %s: %s" % (test.__name__, moerror, warn.message))
+        except Exception as e:
+            success = False
+            print("%s failed on %s: %s" % (test.__name__, moerror, str(e)))
+
+    return success
+
+def testArchive(archive):
+    """Runs all registered tests against all .mo files in the given archive.
+
+       :param str archive: The path to an archive containing .mo files
+       :return: whether the checks succeeded or not
+       :rtype: bool
+    """
+    success = True
+
+    archive_dir = tempfile.mkdtemp(prefix='translation-tests.')
+    try:
+        shutil.unpack_archive(archive, archive_dir)
+        for dir, _dirnames, paths in os.walk(archive_dir):
+            for mofile in (os.path.join(dir, path) for path in paths
+                    if path.endswith('.mo') or path.endswith('.gmo')):
+                if not testFile(mofile, prefix=archive_dir + "/"):
+                    success = False
+    finally:
+        shutil.rmtree(archive_dir, ignore_errors=True)
+
+    return success
diff --git a/translated/__main__.py b/translated/__main__.py
new file mode 100644
index 0000000..fc1ad13
--- /dev/null
+++ b/translated/__main__.py
@@ -0,0 +1,33 @@
+# Entry point for testing translations
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+import sys
+from . import testArchive
+
+if len(sys.argv) < 2:
+    print("Usage: translation_tests <archive> [<archive> ...]")
+    sys.exit(1)
+
+status = 0
+for archive in sys.argv[1:]:
+    if not testArchive(archive):
+        status = 1
+
+sys.exit(status)
diff --git a/translated/test_markup.py b/translated/test_markup.py
new file mode 100644
index 0000000..a3ca417
--- /dev/null
+++ b/translated/test_markup.py
@@ -0,0 +1,44 @@
+# Check translations of pango markup
+#
+# This will look for translatable strings that appear to contain markup and
+# check that the markup in the translation matches.
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+try:
+    import polib
+except ImportError:
+    print("You need to install the python-polib package to read translations")
+    raise
+
+from pocketlint.pangocheck import is_markup, markup_match
+
+def test_markup(mofile):
+    mo = polib.mofile(mofile)
+
+    for entry in mo.translated_entries():
+        if is_markup(entry.msgid):
+            # If this is a plural, check each of the plural translations
+            if entry.msgid_plural:
+                for plural_id, msgstr in entry.msgstr_plural.items():
+                    if not markup_match(entry.msgid, msgstr):
+                        raise AssertionError("Markup does not match for %d translation of msgid %s" %
+                                (plural_id, entry.msgid))
+            elif not markup_match(entry.msgid, entry.msgstr):
+                raise AssertionError("Markup does not match for msgid %s" % entry.msgid)
diff --git a/translated/test_percentage.py b/translated/test_percentage.py
new file mode 100644
index 0000000..914fd09
--- /dev/null
+++ b/translated/test_percentage.py
@@ -0,0 +1,42 @@
+# Check what percentage of strings a .mo translates
+#
+# This will reject translations that fall below a certain threshold of
+# translated strings.
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+import os
+import warnings
+
+try:
+    import polib
+except ImportError:
+    print("You need to install the python-polib package to read translations")
+    raise
+
+threshold = 10
+
+def test_percentage(mofile):
+    # Open the .po file instead, which should be in the same location as the
+    # .mo file in the source archive
+    pofile = polib.pofile(os.path.splitext(mofile)[0] + '.po')
+    if pofile.percent_translated() < threshold:
+        # Issue a warning instead of an exception, since these should probably
+        # be handled on a case-by-case basis
+        warnings.warn("amount translated of %d%% below threshold of %d%%" % (pofile.percent_translated(), threshold))
diff --git a/translated/test_usability.py b/translated/test_usability.py
new file mode 100644
index 0000000..bc51bb9
--- /dev/null
+++ b/translated/test_usability.py
@@ -0,0 +1,28 @@
+# Check a .mo file for basic usability
+#
+# This will test that the file is well-formed and that the Plural-Forms value
+# is parseable
+#
+# Copyright (C) 2015  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): David Shea 
+
+import gettext
+
+def test_usability(mofile):
+    with open(mofile, "rb") as fp:
+        _t = gettext.GNUTranslations(fp=fp)