Multirelease effort: Moving to Python 3

[Nick Coghlan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/22/2013 03:25 PM, Toshio Kuratomi wrote:
> On Mon, Jul 22, 2013 at 10:15:31AM +1000, Nick Coghlan wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>> 
>> On 07/20/2013 06:11 AM, Toshio Kuratomi wrote:
>>> pythonic is a very vague statement and I wouldn't consider most
>>> of your list to be examples of those.  Yes, python3 may be a
>>> *better* language (and I would include most of your list as
>>> "features of python3 that python2 does not have) but a more
>>> pythonic language... that's not something that you can readily
>>> measure.  For instance, I can make the case that python3's
>>> unicode handling is less pythonic than python2 as it violates
>>> three rules of the zen of python:
>>> 
>>> Explicit is better than implicit. Errors should never pass 
>>> silently. In the face of ambiguity, refuse the temptation to 
>>> guess.
>>> 
>>> (To be fair, python2 violated some of these rules in its
>>> unicode handling as well, although errors should never pass
>>> silently would probably take some work to convince most people
>>> :-)
>> 
>> The *only* reason Python 3 allows any Unicode errors to pass
>> silently is because Python 2 tolerated broken system
>> configurations (like non-UTF-8 filesystem metadata on nominally
>> UTF-8 systems) by treating them as opaque 8-bit strings that were
>> retrieved from OS interfaces and then passed back unmodified (see
>> PEP 383 for details). If Python 3 didn't work on those systems,
>> people would blame Python 3, not the already broken system
>> configuration ("But Python 2 works, why is Python 3 so broken?").
>> os.listdir() -> open() is the canonical example of the kind of
>> "round trip" activity that we felt we needed to support even for
>> systems with improperly encoded metadata (including file names).
>> 
> 
> Actually, surrogateescape is a *great* improvement over the
> previous python3 behaviour of silently dropping data that it did
> not understand.

That behaviour only existed in 3.1. It's one of the reasons nobody
really used 3.1 for anything ;)

> If python3 could just finally fix outputting text with
> surrogateescaped bytes then it would finally clean up the last
> portion of this and I would be able to stop pointing out the
> various ways that python3's unicode handling is just as broken as
> pyhton2's -- just in different ways. :-)

Attempting to encode data containing surrogate escapes without setting
"errors=surrogateescape" is a sign that tainted data has escaped
somewhere. So it's late notification of an error, but still indicative
of an error somewhere. We'll never silence it by default.


>> Tainting would involve having the surrogateescape codec set an 
>> attribute on a string recording the encoding assumption if it had
>> to embed any surrogates in the Private Use Area, as well as a
>> keyword only "taint" argument to decode operations (e.g. to force
>> tainting when using "latin-1" as a universal text codec). Various
>> string operations would then be modified to use the following
>> rules:
>> 
>> * Both input strings untainted? Output is untainted. * One input
>> tainted, one untainted? Output is tainted with the same 
>> assumption as the tainted input * Both inputs tainted with the
>> same assumption? Output is also tainted with that assumption.
>> 
> This sounds like it might be nice.  The one thing I'm a little
> unsure about is that it sounds like code is going to have to handle
> this explicitly. Judging from the way all but a select few people
> handle Text vs encoded bytes right now, that seems like it won't
> achieve very much.  OTOH, I could see this as being an additional
> bit of information that's entirely optional whether people use it.
> I think that could be helpful in some cases of debugging.  (OTOH,
> often when encoding vs text issues arise it's because the coder and
> program have no way to know the correct encoding.  When that 
> happens, so the extra information might not be that useful for the
> majority of cases anyway).
> 
>> * Inputs tainted with different assumptions? Immediate
>> ValueError complaining about the taint mismatch
>> 
>> String encoding would be updated to trigger a ValueError when
>> asked to encode tainted strings to an encoding other than the
>> tainted one.
>> 
> 
> I'm a little leery of these.  The reason is that after using both
> python2 and the early versions of python3 I became a firm believer
> that the problem with python2's unicode handling wasn't that it
> threw exceptions, rather the problem was that the same bit of code
> was too prone to passing through certain data without error and
> throwing an error with other data. Programmers who tested their
> code with only ascii data or only data encoded in their locale's
> encoding, or only when their locale was a utf-8 encoding were
> unable to replicate or understand the errors that their user's got
> when they ran them in the crazy real-world environments that user's
> inevitably have.  These rules that throw an Exception suffer from
> the same reliance on the specific data and environment and will
> lead to similar tracebacks that programmers won't be able to easily
> replicate.

You can't get away from this problem: decoding with the wrong encoding
corrupts your data. The only thing we can control are the consequences
of that data corruption. The aim of Python 3 is that the *default*
behaviour will be to produce an exception (hopefully with enough info
to let you debug the situation), but provide various tools to let you
say "I prefer the risk of silent output corruption, thanks."

Once the PEP 432 initialisation model is in place, I'd also like to
provide a "force UTF-8" setting, so people can more easily ensure
cross-platform consistency, even if some of the systems they use
aren't configured to use UTF-8 as the environmental encoding.

Cheers,
Nick.

- -- 
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJR7NwmAAoJEHEkJo9fMO/LirQH/1FBBQM5qiQjN7mzjH7egWr0
ykxjs2Xm1FfAV4VOOUSDnEPU24tJ5yAf+yu3+vDlhiu8hdPXQnZ4xeqlmDOmxHHZ
xnURHrfZl7rZDHvjkaGb4ilYuxYFhUIgeSTNDgrajPxdCP92D54vgtX/Y04PcFWb
ATc24oaj/zYuA7siqwe6YGrtHg6ON93kZvYWgSX+p4x0JIdyjv8brS/LkohyVSkl
6bRHqnmwt3dMx87JaPZwjHhAn2hpNYEYYdac6zAkyT3b2AgHYHYlob0E9bjk+0nk
60p1cSwqK7NTMaGFR8lyt1V7bJsO6z3vGkjKW/kj9BTGmqGDRtVih0xEMw+Pe30=
=FNqT
-----END PGP SIGNATURE-----