1:04 a.m.
Hi team,
At PyCon Australia on the weekend I was reminded of PEP-484 type
hinting** and the Mypy type checker for Python.
With focus of FreeIPA project shifting more towards stability,
quality and maintainability, and with Python 3 porting work nearly
wrapped up, now is the time to think about how we can get more
confidence in our code not just from tests, but from the code
itself. Static checking of annotated types can help us there, and
Mypy can let us begin to do this when writing new code or
refactoring old code. Furthermore there is a benefit for IDE-users
where plugins can use type annotations to provide better completion
suggestions, etc. For an overview of Mypy please see the PyCon AU
talk[1] or the docs[2].
[1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
[2] http://mypy.readthedocs.io/en/latest/index.html
So, what's the plan? Alongside my other tasks, I'm going to start
looking at how we could use Mypy in FreeIPA CI, and see what it is
like using types in some of the areas I'm familiar with e.g.
ipalib.x509. Based on my findings I'll update the team on the wins
and challenges and we can decide how to proceed from there.
Cheers,
Fraser
3:18 a.m.
On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote:
Hi team,
At PyCon Australia on the weekend I was reminded of PEP-484 type
hinting** and the Mypy type checker for Python.
With focus of FreeIPA project shifting more towards stability,
quality and maintainability, and with Python 3 porting work nearly
wrapped up, now is the time to think about how we can get more
confidence in our code not just from tests, but from the code
itself. Static checking of annotated types can help us there, and
Mypy can let us begin to do this when writing new code or
refactoring old code. Furthermore there is a benefit for IDE-users
where plugins can use type annotations to provide better completion
suggestions, etc. For an overview of Mypy please see the PyCon AU
talk[1] or the docs[2].
[1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
[2] http://mypy.readthedocs.io/en/latest/index.html
So, what's the plan? Alongside my other tasks, I'm going to start
looking at how we could use Mypy in FreeIPA CI, and see what it is
like using types in some of the areas I'm familiar with e.g.
ipalib.x509. Based on my findings I'll update the team on the wins
and challenges and we can decide how to proceed from there.
Felipe ask me about typing and Mypy a couple of weeks ago. It's a good
idea and we should do it. But I advise against typing information in the
source code. FreeIPA should use external stub files for two reasons.
First of all it is required to stay compatible with Python 2. And more
importantly it's faster. FreeIPA's CLI scripts already take several
hundred milliseconds to execute. Typing would slow them down even further.
It's rather easy to auto-generate stub files -- assuming you are running
on Fedora and have all Python 3 dependencies installed:
$ sudo dnf install python3-mypy
$ echo "api.bootstrap(ra_plugin='dogtag')" >> ipalib/__init__.py
$ mkdir out
$ PYTHONPATH=. stubgen --recursive ipaclient ipalib ipaplatform
ipapython ipaserver
The api.bootstrap() call is required. Otherwise stubgen cannot import a
bunch of plugin files.
Christian
--
Christian Heimes
Senior Software Engineer, Identity Management and Platform Security
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander
3:37 a.m.
On 2017-08-09 10:18, Christian Heimes via FreeIPA-devel wrote:
On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote:
> Hi team,
>
> At PyCon Australia on the weekend I was reminded of PEP-484 type
> hinting** and the Mypy type checker for Python.
>
> With focus of FreeIPA project shifting more towards stability,
> quality and maintainability, and with Python 3 porting work nearly
> wrapped up, now is the time to think about how we can get more
> confidence in our code not just from tests, but from the code
> itself. Static checking of annotated types can help us there, and
> Mypy can let us begin to do this when writing new code or
> refactoring old code. Furthermore there is a benefit for IDE-users
> where plugins can use type annotations to provide better completion
> suggestions, etc. For an overview of Mypy please see the PyCon AU
> talk[1] or the docs[2].
>
> [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
> [2] http://mypy.readthedocs.io/en/latest/index.html
>
> So, what's the plan? Alongside my other tasks, I'm going to start
> looking at how we could use Mypy in FreeIPA CI, and see what it is
> like using types in some of the areas I'm familiar with e.g.
> ipalib.x509. Based on my findings I'll update the team on the wins
> and challenges and we can decide how to proceed from there.
Felipe ask me about typing and Mypy a couple of weeks ago. It's a good
idea and we should do it. But I advise against typing information in the
source code. FreeIPA should use external stub files for two reasons.
First of all it is required to stay compatible with Python 2. And more
importantly it's faster. FreeIPA's CLI scripts already take several
hundred milliseconds to execute. Typing would slow them down even further.
It's rather easy to auto-generate stub files -- assuming you are running
on Fedora and have all Python 3 dependencies installed:
Small correction:, stubgen in Fedora 26 is broken [1]. Try this:
https://gist.github.com/tiran/a281aa3baf9ea39e1d02800c1a7f8ea6
Christian
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1479697
--
Christian Heimes
Senior Software Engineer, Identity Management and Platform Security
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander
8:26 p.m.
On Wed, Aug 09, 2017 at 10:18:33AM +0200, Christian Heimes via FreeIPA-devel wrote:
On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote:
> Hi team,
>
> At PyCon Australia on the weekend I was reminded of PEP-484 type
> hinting** and the Mypy type checker for Python.
>
> With focus of FreeIPA project shifting more towards stability,
> quality and maintainability, and with Python 3 porting work nearly
> wrapped up, now is the time to think about how we can get more
> confidence in our code not just from tests, but from the code
> itself. Static checking of annotated types can help us there, and
> Mypy can let us begin to do this when writing new code or
> refactoring old code. Furthermore there is a benefit for IDE-users
> where plugins can use type annotations to provide better completion
> suggestions, etc. For an overview of Mypy please see the PyCon AU
> talk[1] or the docs[2].
>
> [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
> [2] http://mypy.readthedocs.io/en/latest/index.html
>
> So, what's the plan? Alongside my other tasks, I'm going to start
> looking at how we could use Mypy in FreeIPA CI, and see what it is
> like using types in some of the areas I'm familiar with e.g.
> ipalib.x509. Based on my findings I'll update the team on the wins
> and challenges and we can decide how to proceed from there.
Felipe ask me about typing and Mypy a couple of weeks ago. It's a good
idea and we should do it. But I advise against typing information in the
source code. FreeIPA should use external stub files for two reasons.
First of all it is required to stay compatible with Python 2. And more
importantly it's faster. FreeIPA's CLI scripts already take several
hundred milliseconds to execute. Typing would slow them down even further.
I disagree with using stub files. Types should be declared where
the functions are defined. Types are documentation and proximity is
important (for humans).
Fortunately, Mypy supports "type comments" in addition to PEP 3107
function annotations. Mypy groks them but CPython will treat them
as comments and discard. This allows us to use type hints with no
runtime cost for the CLI scripts.
It's rather easy to auto-generate stub files -- assuming you are
running
on Fedora and have all Python 3 dependencies installed:
$ sudo dnf install python3-mypy
$ echo "api.bootstrap(ra_plugin='dogtag')" >> ipalib/__init__.py
$ mkdir out
$ PYTHONPATH=. stubgen --recursive ipaclient ipalib ipaplatform
ipapython ipaserver
The api.bootstrap() call is required. Otherwise stubgen cannot import a
bunch of plugin files.
Thanks for this additional info.
Cheers,
Fraser
3:24 a.m.
On 08/10/2017 03:26 AM, Fraser Tweedale via FreeIPA-devel wrote:
On Wed, Aug 09, 2017 at 10:18:33AM +0200, Christian Heimes via
FreeIPA-devel wrote:
> On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote:
>> Hi team,
>>
>> At PyCon Australia on the weekend I was reminded of PEP-484 type
>> hinting** and the Mypy type checker for Python.
>>
>> With focus of FreeIPA project shifting more towards stability,
>> quality and maintainability, and with Python 3 porting work nearly
>> wrapped up, now is the time to think about how we can get more
>> confidence in our code not just from tests, but from the code
>> itself. Static checking of annotated types can help us there, and
>> Mypy can let us begin to do this when writing new code or
>> refactoring old code. Furthermore there is a benefit for IDE-users
>> where plugins can use type annotations to provide better completion
>> suggestions, etc. For an overview of Mypy please see the PyCon AU
>> talk[1] or the docs[2].
>>
>> [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
>> [2] http://mypy.readthedocs.io/en/latest/index.html
>>
>> So, what's the plan? Alongside my other tasks, I'm going to start
>> looking at how we could use Mypy in FreeIPA CI, and see what it is
>> like using types in some of the areas I'm familiar with e.g.
>> ipalib.x509. Based on my findings I'll update the team on the wins
>> and challenges and we can decide how to proceed from there.
> Felipe ask me about typing and Mypy a couple of weeks ago. It's a good
> idea and we should do it. But I advise against typing information in the
> source code. FreeIPA should use external stub files for two reasons.
> First of all it is required to stay compatible with Python 2. And more
> importantly it's faster. FreeIPA's CLI scripts already take several
> hundred milliseconds to execute. Typing would slow them down even further.
>
I disagree with using stub files. Types should be declared where
the functions are defined. Types are documentation and proximity is
important (for humans).
Fortunately, Mypy supports "type comments" in addition to PEP 3107
function annotations. Mypy groks them but CPython will treat them
as comments and discard. This allows us to use type hints with no
runtime cost for the CLI scripts.
I like the idea of specifying types and it can
also be helpful when
reading the code, because it serves as additional documentation. For
this reason, I also think it's important not to use stub files.
The Python 3.6 syntax looks awesome and I'd like to use it. However,
since we have to support Python 2, we'd have to use the comments. I
don't really like that, because it's not a part of the in-line code.
There's also a big downside -- you can't use partial annotations with
comments.
--
Tomas Krizek
PGP: 4A8B A48C 2AED 933B D495 C509 A1FB A5F7 EF8C 4869
3:45 a.m.
On 2017-08-10 03:26, Fraser Tweedale wrote:
On Wed, Aug 09, 2017 at 10:18:33AM +0200, Christian Heimes via
FreeIPA-devel wrote:
> On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote:
>> Hi team,
>>
>> At PyCon Australia on the weekend I was reminded of PEP-484 type
>> hinting** and the Mypy type checker for Python.
>>
>> With focus of FreeIPA project shifting more towards stability,
>> quality and maintainability, and with Python 3 porting work nearly
>> wrapped up, now is the time to think about how we can get more
>> confidence in our code not just from tests, but from the code
>> itself. Static checking of annotated types can help us there, and
>> Mypy can let us begin to do this when writing new code or
>> refactoring old code. Furthermore there is a benefit for IDE-users
>> where plugins can use type annotations to provide better completion
>> suggestions, etc. For an overview of Mypy please see the PyCon AU
>> talk[1] or the docs[2].
>>
>> [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
>> [2] http://mypy.readthedocs.io/en/latest/index.html
>>
>> So, what's the plan? Alongside my other tasks, I'm going to start
>> looking at how we could use Mypy in FreeIPA CI, and see what it is
>> like using types in some of the areas I'm familiar with e.g.
>> ipalib.x509. Based on my findings I'll update the team on the wins
>> and challenges and we can decide how to proceed from there.
>
> Felipe ask me about typing and Mypy a couple of weeks ago. It's a good
> idea and we should do it. But I advise against typing information in the
> source code. FreeIPA should use external stub files for two reasons.
> First of all it is required to stay compatible with Python 2. And more
> importantly it's faster. FreeIPA's CLI scripts already take several
> hundred milliseconds to execute. Typing would slow them down even further.
>
I disagree with using stub files. Types should be declared where
the functions are defined. Types are documentation and proximity is
important (for humans).
>
Fortunately, Mypy supports "type comments" in addition to PEP 3107
function annotations. Mypy groks them but CPython will treat them
as comments and discard. This allows us to use type hints with no
runtime cost for the CLI scripts.
This should be an issue with the right editor. For example PyCharm
supports stub files and displays type annotations. But I'm ok with both
external stub files and full inline comments. Since the FreeIPA team is
going to be both author and consumer of type annotations, the inline
comment style might be more appropriate. Unfortunately inline coments
can't be automated with stubgen. :/
Christian
--
Christian Heimes
Senior Software Engineer, Identity Management and Platform Security
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander
3:49 a.m.
On 2017-08-10 10:24, Tomas Krizek wrote:
I like the idea of specifying types and it can also be helpful when
reading the code, because it serves as additional documentation. For
this reason, I also think it's important not to use stub files.
The Python 3.6 syntax looks awesome and I'd like to use it. However,
since we have to support Python 2, we'd have to use the comments. I
don't really like that, because it's not a part of the in-line code.
There's also a big downside -- you can't use partial annotations with
comments.
I'm sorry for being the bringer of bad news. Although the Python 3
syntax (supported since Python 3.0!) looks awesome, you don't want to
use them in FreeIPA. Trust me. :)
Type annotation have a negative impact on both memory consumption and
import time. After all you end up with thousands of additional Python
objects in memory. The objects have to be constructed, validated and
attached to __annotations__ attribute.
Christian
--
Christian Heimes
Senior Software Engineer, Identity Management and Platform Security
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander
2461
days inactive
2463
days old
freeipa-devel@lists.fedorahosted.org
6 comments
3 participants
participants (3)
-
Christian Heimes -
Fraser Tweedale -
Tomas Krizek