Python libraries problem with F34

[Frederic Muller]

Hello!

Not a developer but I do write code sometimes, and in Python most of the 
times.

Some of my Python code/scripts which were running without any issue 
until F33, no longer do with F34. I actually upgraded from F33 to F34.

I had issues with pexpect (which just now is working again...) and 
requests/urllib. The requests issue is still here and while the code is 
running I get this error:
"warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported
"
RequestsDependencyWarning: urllib3 (1.25.10) or chardet (4.0.0) doesn't 
match a supported version!"

I assume it's a library mismatch but I'm not sure exactly what to do 
about it and my interweb searches did manage to address the issue neither.

Any idea what could be wrong?

Thank you.

Fred

[Felix Schwarz]

- Can you post a small code snippet which reproduces the problem?

- I can use requests on F34 without any warnings. Any chance you manually
   installed other libraries/versions in Fedora's site-packages?
   (/usr/lib64/python3.9)
Felix

[Frederic Muller]

On 7/5/21 2:23 PM, Felix Schwarz wrote:
>  - Can you post a small code snippet which reproduces the problem?
>
>  - I can use requests on F34 without any warnings. Any chance you manually
>    installed other libraries/versions in Fedora's site-packages?
>    (/usr/lib64/python3.9)
>  Felix
>  
Dear Felix,

Sure, it's definitely nothing top secret! ;-)

import os, requests
from urllib.parse import urlparse

# init
username = 'cannotsay'
password = 'willnotshare'

url = 'https://secret.url.com/file.txt'
# Actual downloading
fileons = os.path.basename(urlparse(url).path)
r = requests.get(url, auth=(username,password))
if r.status_code == 200:
     with open(fileons, 'wb') as out:
         for bits in r.iter_content():
             out.write(bits)

I install all my libraries using pip.

Thank you.

Fred

[Felix Schwarz]

Hi Fred,

your code works for me without printing any warnings.

Am 05.07.21 um 10:01 schrieb Frederic Muller:> I install all my libraries using pip.
The important thing is that you never use pip to install anything into Fedora's 
site-packages /usr/lib64/python3.9/site-packages/ . That can (and will) cause a 
lot of trouble. Use "dnf" only to install stuff into your system site-packages 
directory. Use virtualenvs (or similar mechanisms) to install your own libraries 
with pip.

Actually I see a lot of bug reports about certbot in Fedora/EPEL where users 
installed custom Python libraries which break Fedora's certbot. For Python 
packages included in Fedora we can check that everything works fine (though 
sometimes a bug might go unnoticed).


I suggest you revert all manual changes first, reinstall all Python packages and 
then create a custom virtualenv if you need additional libraries:

$ rpm -qf /usr/lib64/python3.9/site-packages/* | grep 'is not owned by any
package'
-> you will see a list of files/directories which are not part of Fedora's
    packages. You should delete those.

Once you did that, check all remaining Python packages which might have been 
replaced by pip:
$ rpm -qf /usr/lib64/python3.9/site-packages/* | xargs rpm --verify

You'll see a list of changed files, e.g.:
S.5....T.    /usr/share/glib-2.0/schemas/gschemas.compiled

If these files are in /usr/lib64/python3.9/site-packages/ you can check which 
python package these belong to ("rpm -qf /path/to/file") and reinstall those 
packages using "dnf reinstall ...".

Once you have done that you Fedora system should be fine again and you can start 
using virtualenvs.

Felix

[Frederic Muller]

On 7/5/21 4:01 PM, Felix Schwarz wrote:
>  Hi Fred,
>
>  your code works for me without printing any warnings.
>
>  Am 05.07.21 um 10:01 schrieb Frederic Muller:> I install all my 
>  libraries using pip.
>  The important thing is that you never use pip to install anything into 
>  Fedora's site-packages /usr/lib64/python3.9/site-packages/ . That can 
>  (and will) cause a lot of trouble. Use "dnf" only to install stuff 
>  into your system site-packages directory. Use virtualenvs (or similar 
>  mechanisms) to install your own libraries with pip. 
Ok I didn't know that. I
however install them for my user. So the (full) 
error message is actually:
Warning (from warnings module):
   File 
"/home/fred/.local/lib/python3.9/site-packages/requests/__init__.py", 
line 89

I guess it's the same as /usr/lib64/ then?
>  Actually I see a lot of bug reports about certbot in Fedora/EPEL where 
>  users installed custom Python libraries which break Fedora's certbot. 
>  For Python packages included in Fedora we can check that everything 
>  works fine (though sometimes a bug might go unnoticed).
>
>
>  I suggest you revert all manual changes first, reinstall all Python 
>  packages and then create a custom virtualenv if you need additional 
>  libraries:
>
>  $ rpm -qf /usr/lib64/python3.9/site-packages/* | grep 'is not owned by 
>  any package'
>  -> you will see a list of files/directories which are not part of 
>  Fedora's
>     packages. You should delete those.
>
>  Once you did that, check all remaining Python packages which might 
>  have been replaced by pip:
>  $ rpm -qf /usr/lib64/python3.9/site-packages/* | xargs rpm --verify
>
>  You'll see a list of changed files, e.g.:
>  S.5....T.    /usr/share/glib-2.0/schemas/gschemas.compiled
>
>  If these files are in /usr/lib64/python3.9/site-packages/ you can 
>  check which python package these belong to ("rpm -qf /path/to/file") 
>  and reinstall those packages using "dnf reinstall ...".
>
>  Once you have done that you Fedora system should be fine again and you 
>  can start using virtualenvs.
>
>  Felix 
Well thank you very much. I know what to do now then. :-)

Fred

[Alexander Bokovoy]

On ma, 05 heinä 2021, Frederic Muller wrote:
> On 7/5/21 4:01 PM, Felix Schwarz wrote:
> >Hi Fred,
> >
> >your code works for me without printing any warnings.
> >
> >Am 05.07.21 um 10:01 schrieb Frederic Muller:> I install all my 
> >libraries using pip.
> >The important thing is that you never use pip to install anything 
> >into Fedora's site-packages /usr/lib64/python3.9/site-packages/ . 
> >That can (and will) cause a lot of trouble. Use "dnf" only to 
> >install stuff into your system site-packages directory. Use 
> >virtualenvs (or similar mechanisms) to install your own libraries 
> >with pip.
> Ok I didn't know that. I however install them for my user. So the 
> (full) error message is actually:
> Warning (from warnings module):
>   File 
> "/home/fred/.local/lib/python3.9/site-packages/requests/__init__.py", 
> line 89
>
> I guess it's the same as /usr/lib64/ then? 
If you want to use pip, make sure you are doing so within a virtual
environment, e.g. with venv or similar constructs. This ensures whatever
was installed is self-consistent with regards to the source where it
came from.



-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland