[PATCH] Utils: allow non-secure md5 use

[Jan Tluka]

This will fix the issue with LNST on FIPS enabled system. FIPS does not allow
use of md5. Python's hashlib has a parameter that can be used to allow use of
md5 for non-security purpose. LNST uses md5 for caching test modules and tools
only.

Tested both with python2.6 and python2.7

Signed-off-by: Jan Tluka <jtluka(a)redhat.com&gt;
---
 lnst/Common/Utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lnst/Common/Utils.py b/lnst/Common/Utils.py
index d6d6c57..1a7c926 100644
--- a/lnst/Common/Utils.py
+++ b/lnst/Common/Utils.py
@@ -98,7 +98,7 @@ def bool_it(val):
     return True if int_it(val) else False
 
 def md5sum(file_path, block_size=2**20):
-    md5 = hashlib.md5()
+    md5 = hashlib.md5(usedforsecurity=False)
     with open(file_path, "rb") as f:
         while True:
             data = f.read(block_size)
-- 
2.7.5

[Ondrej Lichtner]

On Wed, Jun 28, 2017 at 08:13:29PM +0200, Jan Tluka wrote:
>  This will fix the issue with LNST on FIPS enabled system. FIPS does
> not allow
>  use of md5. Python's hashlib has a parameter that can be used to allow use of
>  md5 for non-security purpose. LNST uses md5 for caching test modules and tools
>  only.
>  
>  Tested both with python2.6 and python2.7
>  
>  Signed-off-by: Jan Tluka <jtluka(a)redhat.com&gt;
>  ---
>   lnst/Common/Utils.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>  
>  diff --git a/lnst/Common/Utils.py b/lnst/Common/Utils.py
>  index d6d6c57..1a7c926 100644
>  --- a/lnst/Common/Utils.py
>  +++ b/lnst/Common/Utils.py
>  @@ -98,7 +98,7 @@ def bool_it(val):
>       return True if int_it(val) else False
>   
>   def md5sum(file_path, block_size=2**20):
>  -    md5 = hashlib.md5()
>  +    md5 = hashlib.md5(usedforsecurity=False)
>       with open(file_path, "rb") as f:
>           while True:
>               data = f.read(block_size)
>  -- 
>  2.7.5
>  _______________________________________________
>  LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
>  To unsubscribe send an email to lnst-developers-leave(a)lists.fedorahosted.org 
pushed, thanks.

-Ondrej

[Roi Dayan]

On 04/08/2017 10:08, Ondrej Lichtner wrote:
>  On Wed, Jun 28, 2017 at 08:13:29PM +0200, Jan Tluka wrote:
> > This will fix the issue with LNST on FIPS enabled system. FIPS does not allow
> > use of md5. Python's hashlib has a parameter that can be used to allow use of
> > md5 for non-security purpose. LNST uses md5 for caching test modules and tools
> > only.
> >
> > Tested both with python2.6 and python2.7
> >
> > Signed-off-by: Jan Tluka <jtluka(a)redhat.com&gt;
> > ---
> >  lnst/Common/Utils.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/lnst/Common/Utils.py b/lnst/Common/Utils.py
> > index d6d6c57..1a7c926 100644
> > --- a/lnst/Common/Utils.py
> > +++ b/lnst/Common/Utils.py
> > @@ -98,7 +98,7 @@ def bool_it(val):
> >      return True if int_it(val) else False
> >
> >  def md5sum(file_path, block_size=2**20):
> > -    md5 = hashlib.md5()
> > +    md5 = hashlib.md5(usedforsecurity=False)
> >      with open(file_path, "rb") as f:
> >          while True:
> >              data = f.read(block_size)
> > --
> > 2.7.5
> > _______________________________________________
> > LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
> > To unsubscribe send an email to lnst-developers-leave(a)lists.fedorahosted.org
>
>  pushed, thanks.
>
>  -Ondrej
>  _______________________________________________
>  LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
>  To unsubscribe send an email to lnst-developers-leave(a)lists.fedorahosted.org
>  
Hi,

This actually broke lnst on my machine.
I don't have the usedforsecurity option in python2.7 or python3
but do in python2.6.
I encountered this because I run lnst with python2.7.

root@mtr-stm-023 ~ #  python2.6 -c "import hashlib ; 
hashlib.md5(usedforsecurity=False)"

root@mtr-stm-023 ~ #  python2.7 -c "import hashlib ; 
hashlib.md5(usedforsecurity=False)"
Traceback (most recent call last):
   File "<string>", line 1, in <module>
TypeError: openssl_md5() takes no keyword arguments

root@mtr-stm-023 ~ # 1 python3 -c "import hashlib ; 
hashlib.md5(usedforsecurity=False)"
Traceback (most recent call last):
   File "<string>", line 1, in <module>
TypeError: openssl_md5() takes no keyword arguments
root@mtr-stm-023 ~ # 1


any idea? is it the python build?
maybe there should be a check if the flag exists and only
then use it?

Thanks,
Roi

[Jiri Prochazka]

2017-08-09 17:13 GMT+02:00 Roi Dayan <roid(a)mellanox.com&gt;:

>  On 04/08/2017 10:08, Ondrej Lichtner wrote:
>
> > On Wed, Jun 28, 2017 at 08:13:29PM +0200, Jan Tluka wrote:
> >
> >> This will fix the issue with LNST on FIPS enabled system. FIPS does not
> >> allow
> >> use of md5. Python's hashlib has a parameter that can be used to allow
> >> use of
> >> md5 for non-security purpose. LNST uses md5 for caching test modules and
> >> tools
> >> only.
> >>
> >> Tested both with python2.6 and python2.7
> >>
> >> Signed-off-by: Jan Tluka <jtluka(a)redhat.com&gt;
> >> ---
> >>  lnst/Common/Utils.py | 2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/lnst/Common/Utils.py b/lnst/Common/Utils.py
> >> index d6d6c57..1a7c926 100644
> >> --- a/lnst/Common/Utils.py
> >> +++ b/lnst/Common/Utils.py
> >> @@ -98,7 +98,7 @@ def bool_it(val):
> >>      return True if int_it(val) else False
> >>
> >>  def md5sum(file_path, block_size=2**20):
> >> -    md5 = hashlib.md5()
> >> +    md5 = hashlib.md5(usedforsecurity=False)
> >>      with open(file_path, "rb") as f:
> >>          while True:
> >>              data = f.read(block_size)
> >> --
> >> 2.7.5
> >> _______________________________________________
> >> LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
> >> To unsubscribe send an email to lnst-developers-leave(a)lists.fe
> >> dorahosted.org
> >>
> >
> > pushed, thanks.
> >
> > -Ondrej
> > _______________________________________________
> > LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
> > To unsubscribe send an email to lnst-developers-leave(a)lists.fe
> > dorahosted.org
> >
> >
>  Hi,
>
>  This actually broke lnst on my machine.
>  I don't have the usedforsecurity option in python2.7 or python3
>  but do in python2.6.
>  I encountered this because I run lnst with python2.7.
>
>  root@mtr-stm-023 ~ #  python2.6 -c "import hashlib ;
>  hashlib.md5(usedforsecurity=False)"
>
>  root@mtr-stm-023 ~ #  python2.7 -c "import hashlib ;
>  hashlib.md5(usedforsecurity=False)"
>  Traceback (most recent call last):
>    File "<string>", line 1, in <module>
>  TypeError: openssl_md5() takes no keyword arguments
>  
​Hi,

no traceback for me on Python 2.7.13, everything works fine.​



>  root@mtr-stm-023 ~ # 1 python3 -c "import hashlib ;
>  hashlib.md5(usedforsecurity=False)"
>  Traceback (most recent call last):
>    File "<string>", line 1, in <module>
>  TypeError: openssl_md5() takes no keyword arguments
>  root@mtr-stm-023 ~ # 1
>
>
>  any idea? is it the python build?
>  maybe there should be a check if the flag exists and only
>  then use it?
>
>  Thanks,
>  Roi
>
>  _______________________________________________
>  LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
>  To unsubscribe send an email to lnst-developers-leave(a)lists.fe
>  dorahosted.org
>

[Ondrej Lichtner]

On Wed, Aug 09, 2017 at 06:13:51PM +0300, Roi Dayan wrote:
>  Hi,
>  
>  This actually broke lnst on my machine.
>  I don't have the usedforsecurity option in python2.7 or python3
>  but do in python2.6.
>  I encountered this because I run lnst with python2.7.
>  
>  root@mtr-stm-023 ~ #  python2.6 -c "import hashlib ;
>  hashlib.md5(usedforsecurity=False)"
>  
>  root@mtr-stm-023 ~ #  python2.7 -c "import hashlib ;
>  hashlib.md5(usedforsecurity=False)"
>  Traceback (most recent call last):
>    File "<string>", line 1, in <module>
>  TypeError: openssl_md5() takes no keyword arguments
>  
>  root@mtr-stm-023 ~ # 1 python3 -c "import hashlib ;
>  hashlib.md5(usedforsecurity=False)"
>  Traceback (most recent call last):
>    File "<string>", line 1, in <module>
>  TypeError: openssl_md5() takes no keyword arguments
>  root@mtr-stm-023 ~ # 1
>  
>  
>  any idea? is it the python build?
>  maybe there should be a check if the flag exists and only
>  then use it?
>  
>  Thanks,
>  Roi 
Hi,

interesting, it seems that the "usedforsecurity" patch is something
applied downstream by the Fedora packagers and it wasn't accepted into
upstream python yet [1]. The issue is already 7 years old so let's not
get our hopes up for a quick resolution...

So for LNST I see two possibilities:
* move to a newer hash that fips doesn't disable - I already do this on
  the 'next' branch and we only use md5 for file/directory hashing
  making the required changes pretty contained.
  * Impact on running LNST instances will be that all test resources will
    be resynchronized again but that's not a big problem.
  * This technically isn't a futureproof solution... fips will inevitably
    change and sha256 will be deprecated when we'll encounter the same
    issues... But it might be enough to postpone the issue until python
    upstream solves this properly.
* implement the other proposed solution (which I think I only discussed
  with jtluka in person) and just catch the proper exceptions - either
  the "fips forbidden" exception or the "takes no keyword arguments"
  exception
  * in the end i don't think it solves the issue at all - we still need
    a hash for the file, how do we calculate it if we can't use the
    hashlib function? (running without "usedforsecurity=false" might
    still encounter the "fips forbidden" exception on fips enabled
    systems)

So only one of these really "solves" the issue, by postponing it... does
anyone see a better option than just moving to sha256?

-Ondrej

[1] https://bugs.python.org/issue9216

[Jan Tluka]

Thu, Aug 10, 2017 at 08:47:43AM CEST, olichtne(a)redhat.com wrote:
> On Wed, Aug 09, 2017 at 06:13:51PM +0300, Roi Dayan wrote:
> > Hi,
> > 
> > This actually broke lnst on my machine.
> > I don't have the usedforsecurity option in python2.7 or python3
> > but do in python2.6.
> > I encountered this because I run lnst with python2.7.
> > 
> > root@mtr-stm-023 ~ #  python2.6 -c "import hashlib ;
> > hashlib.md5(usedforsecurity=False)"
> > 
> > root@mtr-stm-023 ~ #  python2.7 -c "import hashlib ;
> > hashlib.md5(usedforsecurity=False)"
> > Traceback (most recent call last):
> >   File "<string>", line 1, in <module>
> > TypeError: openssl_md5() takes no keyword arguments
> > 
> > root@mtr-stm-023 ~ # 1 python3 -c "import hashlib ;
> > hashlib.md5(usedforsecurity=False)"
> > Traceback (most recent call last):
> >   File "<string>", line 1, in <module>
> > TypeError: openssl_md5() takes no keyword arguments
> > root@mtr-stm-023 ~ # 1
> > 
> > 
> > any idea? is it the python build?
> > maybe there should be a check if the flag exists and only
> > then use it?
> > 
> > Thanks,
> > Roi
>
> Hi,
>
> interesting, it seems that the "usedforsecurity" patch is something
> applied downstream by the Fedora packagers and it wasn't accepted into
> upstream python yet [1]. The issue is already 7 years old so let's not
> get our hopes up for a quick resolution...
>
> So for LNST I see two possibilities:
> * move to a newer hash that fips doesn't disable - I already do this on
>   the 'next' branch and we only use md5 for file/directory hashing
>   making the required changes pretty contained.
>   * Impact on running LNST instances will be that all test resources will
>     be resynchronized again but that's not a big problem.
>   * This technically isn't a futureproof solution... fips will inevitably
>     change and sha256 will be deprecated when we'll encounter the same
>     issues... But it might be enough to postpone the issue until python
>     upstream solves this properly.
> * implement the other proposed solution (which I think I only discussed
>   with jtluka in person) and just catch the proper exceptions - either
>   the "fips forbidden" exception or the "takes no keyword arguments"
>   exception
>   * in the end i don't think it solves the issue at all - we still need
>     a hash for the file, how do we calculate it if we can't use the
>     hashlib function? (running without "usedforsecurity=false" might
>     still encounter the "fips forbidden" exception on fips enabled
>     systems)
>
> So only one of these really "solves" the issue, by postponing it... does
> anyone see a better option than just moving to sha256?
>
> -Ondrej
>
> [1] https://bugs.python.org/issue9216 
IMO, the first action is that we should revert the patch if the solution is
not in standard Python. That would make LNST working for everyone until
we fix this properly.

Thoughts?

-Jan

[Roi Dayan]

On 10/08/2017 14:17, Jan Tluka wrote:
>  Thu, Aug 10, 2017 at 08:47:43AM CEST, olichtne(a)redhat.com wrote:
> > On Wed, Aug 09, 2017 at 06:13:51PM +0300, Roi Dayan wrote:
> >> Hi,
> >>
> >> This actually broke lnst on my machine.
> >> I don't have the usedforsecurity option in python2.7 or python3
> >> but do in python2.6.
> >> I encountered this because I run lnst with python2.7.
> >>
> >> root@mtr-stm-023 ~ #  python2.6 -c "import hashlib ;
> >> hashlib.md5(usedforsecurity=False)"
> >>
> >> root@mtr-stm-023 ~ #  python2.7 -c "import hashlib ;
> >> hashlib.md5(usedforsecurity=False)"
> >> Traceback (most recent call last):
> >>   File "<string>", line 1, in <module>
> >> TypeError: openssl_md5() takes no keyword arguments
> >>
> >> root@mtr-stm-023 ~ # 1 python3 -c "import hashlib ;
> >> hashlib.md5(usedforsecurity=False)"
> >> Traceback (most recent call last):
> >>   File "<string>", line 1, in <module>
> >> TypeError: openssl_md5() takes no keyword arguments
> >> root@mtr-stm-023 ~ # 1
> >>
> >>
> >> any idea? is it the python build?
> >> maybe there should be a check if the flag exists and only
> >> then use it?
> >>
> >> Thanks,
> >> Roi
> >
> > Hi,
> >
> > interesting, it seems that the "usedforsecurity" patch is something
> > applied downstream by the Fedora packagers and it wasn't accepted into
> > upstream python yet [1]. The issue is already 7 years old so let's not
> > get our hopes up for a quick resolution...
> >
> > So for LNST I see two possibilities:
> > * move to a newer hash that fips doesn't disable - I already do this on
> >  the 'next' branch and we only use md5 for file/directory hashing
> >  making the required changes pretty contained.
> >  * Impact on running LNST instances will be that all test resources will
> >    be resynchronized again but that's not a big problem.
> >  * This technically isn't a futureproof solution... fips will inevitably
> >    change and sha256 will be deprecated when we'll encounter the same
> >    issues... But it might be enough to postpone the issue until python
> >    upstream solves this properly.
> > * implement the other proposed solution (which I think I only discussed
> >  with jtluka in person) and just catch the proper exceptions - either
> >  the "fips forbidden" exception or the "takes no keyword
> arguments"
> >  exception
> >  * in the end i don't think it solves the issue at all - we still need
> >    a hash for the file, how do we calculate it if we can't use the
> >    hashlib function? (running without "usedforsecurity=false" might
> >    still encounter the "fips forbidden" exception on fips enabled
> >    systems)
> >
> > So only one of these really "solves" the issue, by postponing it... does
> > anyone see a better option than just moving to sha256?
> >
> > -Ondrej
> >
> > [1]
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.p...
>
>  IMO, the first action is that we should revert the patch if the solution is
>  not in standard Python. That would make LNST working for everyone until
>  we fix this properly.
>
>  Thoughts?
>
>  -Jan
>  
I'm in favor to remove as it's not standard.
when there will be a standard fix then it wouldn't be a problem to
update a problematic setup.

[Ondrej Lichtner]

On Thu, Aug 10, 2017 at 02:20:04PM +0300, Roi Dayan wrote:
>  
>  
>  On 10/08/2017 14:17, Jan Tluka wrote:
>  > Thu, Aug 10, 2017 at 08:47:43AM CEST, olichtne(a)redhat.com wrote:
>  > > On Wed, Aug 09, 2017 at 06:13:51PM +0300, Roi Dayan wrote:
>  > > > Hi,
>  > > > 
>  > > > This actually broke lnst on my machine.
>  > > > I don't have the usedforsecurity option in python2.7 or python3
>  > > > but do in python2.6.
>  > > > I encountered this because I run lnst with python2.7.
>  > > > 
>  > > > root@mtr-stm-023 ~ #  python2.6 -c "import hashlib ;
>  > > > hashlib.md5(usedforsecurity=False)"
>  > > > 
>  > > > root@mtr-stm-023 ~ #  python2.7 -c "import hashlib ;
>  > > > hashlib.md5(usedforsecurity=False)"
>  > > > Traceback (most recent call last):
>  > > >   File "<string>", line 1, in <module>
>  > > > TypeError: openssl_md5() takes no keyword arguments
>  > > > 
>  > > > root@mtr-stm-023 ~ # 1 python3 -c "import hashlib ;
>  > > > hashlib.md5(usedforsecurity=False)"
>  > > > Traceback (most recent call last):
>  > > >   File "<string>", line 1, in <module>
>  > > > TypeError: openssl_md5() takes no keyword arguments
>  > > > root@mtr-stm-023 ~ # 1
>  > > > 
>  > > > 
>  > > > any idea? is it the python build?
>  > > > maybe there should be a check if the flag exists and only
>  > > > then use it?
>  > > > 
>  > > > Thanks,
>  > > > Roi
>  > > 
>  > > Hi,
>  > > 
>  > > interesting, it seems that the "usedforsecurity" patch is something
>  > > applied downstream by the Fedora packagers and it wasn't accepted into
>  > > upstream python yet [1]. The issue is already 7 years old so let's not
>  > > get our hopes up for a quick resolution...
>  > > 
>  > > So for LNST I see two possibilities:
>  > > * move to a newer hash that fips doesn't disable - I already do this on
>  > >  the 'next' branch and we only use md5 for file/directory hashing
>  > >  making the required changes pretty contained.
>  > >  * Impact on running LNST instances will be that all test resources will
>  > >    be resynchronized again but that's not a big problem.
>  > >  * This technically isn't a futureproof solution... fips will inevitably
>  > >    change and sha256 will be deprecated when we'll encounter the same
>  > >    issues... But it might be enough to postpone the issue until python
>  > >    upstream solves this properly.
>  > > * implement the other proposed solution (which I think I only discussed
>  > >  with jtluka in person) and just catch the proper exceptions - either
>  > >  the "fips forbidden" exception or the "takes no keyword
> arguments"
>  > >  exception
>  > >  * in the end i don't think it solves the issue at all - we still need
>  > >    a hash for the file, how do we calculate it if we can't use the
>  > >    hashlib function? (running without "usedforsecurity=false" might
>  > >    still encounter the "fips forbidden" exception on fips enabled
>  > >    systems)
>  > > 
>  > > So only one of these really "solves" the issue, by postponing it...
> does
>  > > anyone see a better option than just moving to sha256?
>  > > 
>  > > -Ondrej
>  > > 
>  > > [1]
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.p...
>  > 
>  > IMO, the first action is that we should revert the patch if the solution is
>  > not in standard Python. That would make LNST working for everyone until
>  > we fix this properly.
>  > 
>  > Thoughts?
>  > 
>  > -Jan
>  > 
>  
>  I'm in favor to remove as it's not standard.
>  when there will be a standard fix then it wouldn't be a problem to
>  update a problematic setup. 
Ok, agreed, I'll revert tomorrow.

-Ondrej