[PATCH] Add contrib scripts and relevant documentation

[Arun Babu Neelicattu]

From: Arun Babu Neelicattu <abn(a)redhat.com&gt;

---
 .gitignore               |  3 +++
 HACKING                  | 47 +++++++++++++++++++++++++++++++------
 contrib/activate-dev-env | 60 ++++++++++++++++++++++++++++++++++++++++++++++++
 contrib/run-tests        | 21 +++++++++++++++++
 setup.py                 |  2 +-
 5 files changed, 125 insertions(+), 8 deletions(-)
 create mode 100644 contrib/activate-dev-env
 create mode 100755 contrib/run-tests

diff --git a/.gitignore b/.gitignore
index 7608296..8aceef6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,3 +4,6 @@ MANIFEST
 dist
 build
 .coverage
+
+#python-bugzilla venvs
+dev-env*
diff --git a/HACKING b/HACKING
index d269b33..54b77c0 100644
--- a/HACKING
+++ b/HACKING
@@ -1,23 +1,56 @@
-If submitting any patch, please verify that no new pylint or pep8 violations
-are introduced:
+Hackers Guide
+=============
 
-  python setup.py pylint
+1. Setting up the environment
+-----------------------------
+To start development, you can set-up/activate a virtual environment by using the 
+following command:
+    source contrib/activate-dev-env [python2|python3]
 
-And ensure that the basic unit test suite does not regress
+Note: Providing no arguments will attempt to use python2.
 
-  python setup.py test
+# Manually activating an environment
+    source dev-env-${NAME}/bin/activate
 
+2. Running tests
+----------------
+Once you have already activated an environment, you can use the following.
+
+# Basic unit test suite
+    python setup.py test
+
+# Functional tests
 There are more comprehensive tests that are disabled by default. Readonly
 functional tests that run against several public bugzilla instances. No
 login account is required:
 
-  python setup.py test --ro-functional
+    python setup.py test --ro-functional
 
 And read/write functional tests. These currently run against the test
 bugzilla instance at partner-bugzilla.redhat.com, and requires a valid
 login there:
 
-  python setup.py test --rw-functional
+    python setup.py test --rw-functional
+
+Note: Before running rw-functional tests, make sure you have logged using:
+    python bugzilla-cli \
+        --bugzilla="https://partner-bugzilla.redhat.com/xmlrpc.cgi" \
+        --user=$USER login
+
+3. pylint and pep8
+------------------
+To test for pylint or pep8 violations, you can run:
+    python setup.py pylint
+
+Note: This expects that you already have pylint and pep8 (installed when setting
+up virtualenv) isntalled.
+
+4. Patch Submission
+-------------------
+If you are submitting a patch, ensure the following:
+    [REQ] verify that no new pylint or pep8 violations
+    [REQ] run basic unit test suite across all python versions:
+            bash contrib/run-tests
 
 Running any of the functional tests is not a requirement for patch submission,
 but please give them a go if you are interested.
diff --git a/contrib/activate-dev-env b/contrib/activate-dev-env
new file mode 100644
index 0000000..5fcd430
--- /dev/null
+++ b/contrib/activate-dev-env
@@ -0,0 +1,60 @@
+# Helper script to create (if required) and activate a virtual environment for 
+# a specified PYTHON_BINARY (python|python2|python3) and install dependencies 
+# required for testing and development.
+#
+# Usage: source $0 [PYTHON_BINARY]
+
+# use env python by default
+PYTHON_BIN=python2
+VIRTIALENV_CMD=virtualenv
+SETUP_PY="setup.py"
+DEV_MODULE_DIR="bugzilla"
+DIR_PREFIX="dev-env"
+
+if [ $# -ne 0 ]; then
+    PYTHON_BIN=$1
+fi
+
+function activate()
+{
+    DIR_NAME=$1
+    echo "INFO: Activating virtualenv at ${DIR_NAME}"
+    source ${DIR_NAME}/bin/activate
+}
+
+function virtualize()
+{
+    PROMPT=$(basename ${PYTHON_BIN})
+    DIR_NAME="${DIR_PREFIX}-${PROMPT}"
+    
+    if [ ! -d ${DIR_NAME} ]; then
+        echo "INFO: Creating virtualenv ${DIR_NAME}"
+        ${VIRTIALENV_CMD} -p $(which ${PYTHON_BIN}) --prompt "${PROMPT}"
${DIR_NAME}
+    fi
+
+    if [ -d ${DIR_NAME} ]; then
+        activate $DIR_NAME
+        
+        for REQ in $(find ./ -maxdepth 1 -type f -name "*requirements.txt");
do
+            pip install -qr "${REQ}"
+        done
+    else
+        echo >&2 "ERROR: Failed to activate virtualenv."
+    fi
+}
+
+
+if ! command -v ${VIRTIALENV_CMD} >/dev/null 2>&1; then
+    # We require virtuaenv to be available
+    echo >&2 "ERROR: virtualenv command not found"
+elif ! command -v $PYTHON_BIN >/dev/null 2>&1; then
+    # Ensure PYTHON_BIN is valid
+    echo >&2 "ERROR: ${PYTHON_BIN} command not found"
+elif ! ${PYTHON_BIN} --version 2>&1 | grep -q '^Python'; then
+    # Ensure PYTHON_BIN is python
+    echo >&2 "ERROR: '${PYTHON_BIN} --version' returned an invalid
response. Is ${PYTHON_BIN} really python?"
+elif [ ! -f ${SETUP_PY} ]; then
+    echo >&2 "ERROR: ${SETUP_PY} not found"
+else
+    virtualize
+fi
diff --git a/contrib/run-tests b/contrib/run-tests
new file mode 100755
index 0000000..644f5d9
--- /dev/null
+++ b/contrib/run-tests
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+# Helper script to run test for each supported python versions
+# Defaults to running basic tests. The script activates the correct environment
+# as required.
+
+# Usage: $0 [options]
+# Example: $0 --ro-functional
+
+ARGS="$@"
+
+for BIN in python2 python3; do
+    if command -v ${BIN} >/dev/null 2>&1; then
+        echo "INFO: Running tests for ${BIN}"
+        . ./contrib/activate-dev-env ${BIN}
+        python setup.py test ${ARGS}
+        echo "INFO: Tests completed for ${BIN}"
+    else
+        echo "INFO: ${BIN} not found, skipping tests." 
+    fi
+done
diff --git a/setup.py b/setup.py
index 415eee0..6abd9b0 100755
--- a/setup.py
+++ b/setup.py
@@ -49,7 +49,7 @@ class TestCommand(Command):
             usecov = False
 
         if usecov:
-            cov = coverage.coverage(omit=["/*/tests/*", "/usr/*"])
+            cov = coverage.coverage(omit=["/*/tests/*", "/usr/*",
"*dev-env*"])
             cov.erase()
             cov.start()
 
-- 
1.8.3.1

[PATCH] Add contrib scripts and relevant

[Cole Robinson]

>  From: Arun Babu Neelicattu <abn at redhat.com>
>  
>  ---
>   .gitignore               |  3 +++
>   HACKING                  | 47 +++++++++++++++++++++++++++++++------
>   contrib/activate-dev-env | 60 ++++++++++++++++++++++++++++++++++++++++++++++++
>   contrib/run-tests        | 21 +++++++++++++++++
>   setup.py                 |  2 +-
>   5 files changed, 125 insertions(+), 8 deletions(-)
>   create mode 100644 contrib/activate-dev-env
>   create mode 100755 contrib/run-tests 
Thanks, I've pushed this now. But I'm hitting an issue with contrib/run-tests.
The python2 virtualenv setup works, but the python3 one fails:

INFO: Running tests for python3
INFO: Creating virtualenv dev-env-python3
Running virtualenv with interpreter /usr/bin/python3
New python executable in dev-env-python3/bin/python3
Also creating executable in dev-env-python3/bin/python
A globally installed setuptools was found (in /usr/lib/python3.3/site-packages)
Use the --no-site-packages option to use distribute in the virtualenv.
Installing
distribute..................................................................................................................................................................................................................................................................................................................................................................................................done.
Installing pip....
  Complete output from command /home/crobinso/src/p...-python3/bin/python3
/home/crobinso/src/p...on3/bin/easy_install
/usr/lib/python2.7/s...ort/pip-1.0.2.tar.gz:
  /home/crobinso/src/python-bugzilla/dev-env-python3/bin/python3: can't open
file '/home/crobinso/src/python-bugzilla/dev-env-python3/bin/easy_install':
[Errno 2] No such file or directory
----------------------------------------
...Installing pip...done.
Traceback (most recent call last):
  File
"/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
line 1952, in <module>
    main()
  File
"/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
line 810, in main
    never_download=options.never_download)
  File
"/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
line 912, in create_environment
    install_pip(py_executable, search_dirs=search_dirs,
never_download=never_download)
  File
"/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
line 648, in install_pip
    filter_stdout=_filter_setup)
  File
"/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
line 878, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/crobinso/src/p...-python3/bin/python3
/home/crobinso/src/p...on3/bin/easy_install
/usr/lib/python2.7/s...ort/pip-1.0.2.tar.gz failed with error code 2
INFO: Activating virtualenv at dev-env-python3
./contrib/activate-dev-env: line 24: dev-env-python3/bin/activate: No such
file or directory

Any ideas?

- Cole

[PATCH] Add contrib scripts and relevant

[Arun Babu Neelicattu]

hmm, hard to say for sure however, I reckon it has got something to do with the pip
version available

It seems to be using the source for pip from:
>  /usr/lib/python2.7/s...ort/pip-1.0.2.tar.gz 
Not sure if this is the issue, but it could be that the 1.0.2 version was not python 3
ready?

Also I noted this change in the activate-dev-env script:
-            pip install -qr "${REQ}"
+            pip install -qr "${REQ}" || exit 1

That might not be a good idea as we are sourcing the script. This will kill the shell.

-arun

----- Original Message -----
>  From: "Cole Robinson" <crobinso(a)redhat.com&gt;
>  To: "Arun Neelicattu" <abn(a)redhat.com&gt;, "python-bugzilla
> user/developer list"
>  <python-bugzilla(a)lists.fedorahosted.org&gt;
>  Sent: Sunday, November 10, 2013 5:19:16 AM
>  Subject: Re: [python-bugzilla] [PATCH] Add contrib scripts and relevant
>  
>  > From: Arun Babu Neelicattu <abn at redhat.com>
>  > 
>  > ---
>  >  .gitignore               |  3 +++
>  >  HACKING                  | 47 +++++++++++++++++++++++++++++++------
>  >  contrib/activate-dev-env | 60
>  >  ++++++++++++++++++++++++++++++++++++++++++++++++
>  >  contrib/run-tests        | 21 +++++++++++++++++
>  >  setup.py                 |  2 +-
>  >  5 files changed, 125 insertions(+), 8 deletions(-)
>  >  create mode 100644 contrib/activate-dev-env
>  >  create mode 100755 contrib/run-tests
>  
>  Thanks, I've pushed this now. But I'm hitting an issue with
>  contrib/run-tests.
>  The python2 virtualenv setup works, but the python3 one fails:
>  
>  INFO: Running tests for python3
>  INFO: Creating virtualenv dev-env-python3
>  Running virtualenv with interpreter /usr/bin/python3
>  New python executable in dev-env-python3/bin/python3
>  Also creating executable in dev-env-python3/bin/python
>  A globally installed setuptools was found (in
>  /usr/lib/python3.3/site-packages)
>  Use the --no-site-packages option to use distribute in the virtualenv.
>  Installing
>
> distribute..................................................................................................................................................................................................................................................................................................................................................................................................done.
>  Installing pip....
>    Complete output from command /home/crobinso/src/p...-python3/bin/python3
>  /home/crobinso/src/p...on3/bin/easy_install
>  /usr/lib/python2.7/s...ort/pip-1.0.2.tar.gz:
>    /home/crobinso/src/python-bugzilla/dev-env-python3/bin/python3: can't open
>  file '/home/crobinso/src/python-bugzilla/dev-env-python3/bin/easy_install':
>  [Errno 2] No such file or directory
>  ----------------------------------------
>  ...Installing pip...done.
>  Traceback (most recent call last):
>    File
>  "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
>  line 1952, in <module>
>      main()
>    File
>  "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
>  line 810, in main
>      never_download=options.never_download)
>    File
>  "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
>  line 912, in create_environment
>      install_pip(py_executable, search_dirs=search_dirs,
>  never_download=never_download)
>    File
>  "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
>  line 648, in install_pip
>      filter_stdout=_filter_setup)
>    File
>  "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
>  line 878, in call_subprocess
>      % (cmd_desc, proc.returncode))
>  OSError: Command /home/crobinso/src/p...-python3/bin/python3
>  /home/crobinso/src/p...on3/bin/easy_install
>  /usr/lib/python2.7/s...ort/pip-1.0.2.tar.gz failed with error code 2
>  INFO: Activating virtualenv at dev-env-python3
>  ./contrib/activate-dev-env: line 24: dev-env-python3/bin/activate: No such
>  file or directory
>  
>  Any ideas?
>  
>  - Cole
>

[PATCH] Add contrib scripts and relevant

[Cole Robinson]

On 11/10/2013 06:44 PM, Arun Neelicattu wrote:
>  hmm, hard to say for sure however, I reckon it has got something to
> do with the pip version available
>  
>  It seems to be using the source for pip from:
> > /usr/lib/python2.7/s...ort/pip-1.0.2.tar.gz
>  
>  Not sure if this is the issue, but it could be that the 1.0.2 version was not python 3
> ready?
>   
Ah, good call. I had some really old cache there, you can tell I don't use
pip/virtualenv much :)

>  Also I noted this change in the activate-dev-env script:
>  -            pip install -qr "${REQ}"
>  +            pip install -qr "${REQ}" || exit 1
>  
>  That might not be a good idea as we are sourcing the script. This will kill the shell.
>   
Without that, when the python3 init failed, the script continued on and ran
the tests as usual, which all pass since we were still running in the python2
env, which was quite misleading. There may be a better place to stick the
'exit 1' or similar.

Thanks,
Cole

>  
>  ----- Original Message -----
> > From: "Cole Robinson" <crobinso(a)redhat.com&gt;
> > To: "Arun Neelicattu" <abn(a)redhat.com&gt;, "python-bugzilla
> user/developer list"
> > <python-bugzilla(a)lists.fedorahosted.org&gt;
> > Sent: Sunday, November 10, 2013 5:19:16 AM
> > Subject: Re: [python-bugzilla] [PATCH] Add contrib scripts and relevant
> >
> >> From: Arun Babu Neelicattu <abn at redhat.com>
> >>
> >> ---
> >>  .gitignore               |  3 +++
> >>  HACKING                  | 47 +++++++++++++++++++++++++++++++------
> >>  contrib/activate-dev-env | 60
> >>  ++++++++++++++++++++++++++++++++++++++++++++++++
> >>  contrib/run-tests        | 21 +++++++++++++++++
> >>  setup.py                 |  2 +-
> >>  5 files changed, 125 insertions(+), 8 deletions(-)
> >>  create mode 100644 contrib/activate-dev-env
> >>  create mode 100755 contrib/run-tests
> >
> > Thanks, I've pushed this now. But I'm hitting an issue with
> > contrib/run-tests.
> > The python2 virtualenv setup works, but the python3 one fails:
> >
> > INFO: Running tests for python3
> > INFO: Creating virtualenv dev-env-python3
> > Running virtualenv with interpreter /usr/bin/python3
> > New python executable in dev-env-python3/bin/python3
> > Also creating executable in dev-env-python3/bin/python
> > A globally installed setuptools was found (in
> > /usr/lib/python3.3/site-packages)
> > Use the --no-site-packages option to use distribute in the virtualenv.
> > Installing
> >
> distribute..................................................................................................................................................................................................................................................................................................................................................................................................done.
> > Installing pip....
> >   Complete output from command /home/crobinso/src/p...-python3/bin/python3
> > /home/crobinso/src/p...on3/bin/easy_install
> > /usr/lib/python2.7/s...ort/pip-1.0.2.tar.gz:
> >   /home/crobinso/src/python-bugzilla/dev-env-python3/bin/python3: can't open
> > file '/home/crobinso/src/python-bugzilla/dev-env-python3/bin/easy_install':
> > [Errno 2] No such file or directory
> > ----------------------------------------
> > ...Installing pip...done.
> > Traceback (most recent call last):
> >   File
> >
> "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
> > line 1952, in <module>
> >     main()
> >   File
> >
> "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
> > line 810, in main
> >     never_download=options.never_download)
> >   File
> >
> "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
> > line 912, in create_environment
> >     install_pip(py_executable, search_dirs=search_dirs,
> > never_download=never_download)
> >   File
> >
> "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
> > line 648, in install_pip
> >     filter_stdout=_filter_setup)
> >   File
> >
> "/usr/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg/virtualenv.py",
> > line 878, in call_subprocess
> >     % (cmd_desc, proc.returncode))
> > OSError: Command /home/crobinso/src/p...-python3/bin/python3
> > /home/crobinso/src/p...on3/bin/easy_install
> > /usr/lib/python2.7/s...ort/pip-1.0.2.tar.gz failed with error code 2
> > INFO: Activating virtualenv at dev-env-python3
> > ./contrib/activate-dev-env: line 24: dev-env-python3/bin/activate: No such
> > file or directory
> >
> > Any ideas?
> >
> > - Cole
> >

[PATCH] Add contrib scripts and relevant

[Arun Babu Neelicattu]

On Tue, 2013-11-12 at 09:33 -0500, Cole Robinson wrote:
>  
>  Ah, good call. I had some really old cache there, you can tell I don't use
>  pip/virtualenv much :)
>   
:)

>  
>  Without that, when the python3 init failed, the script continued on and ran
>  the tests as usual, which all pass since we were still running in the python2
>  env, which was quite misleading. There may be a better place to stick the
>  'exit 1' or similar.
>   
Hmm, might need to move the check elsewhere, since the run-tests script
is running in a sub-shell it should be file, but this will cause issues
if a we are activating the environment manually. 

Will submit a patch soon. :)

[PATCH] Add contrib scripts and relevant

[Arun Babu Neelicattu]

Sent a patch for this one. Tests should skip if the env is not active now.

----- Original Message -----
>  From: "Arun Babu Neelicattu" <abn(a)redhat.com&gt;
>  To: "Cole Robinson" <crobinso(a)redhat.com&gt;
>  Cc: "python-bugzilla user/developer list"
> <python-bugzilla(a)lists.fedorahosted.org&gt;
>  Sent: Thursday, November 14, 2013 11:38:17 AM
>  Subject: Re: [python-bugzilla] [PATCH] Add contrib scripts and relevant
>  
>  
>  
>  On Tue, 2013-11-12 at 09:33 -0500, Cole Robinson wrote:
>  > 
>  > Ah, good call. I had some really old cache there, you can tell I don't use
>  > pip/virtualenv much :)
>  > 
>  
>  :)
>  
>  > 
>  > Without that, when the python3 init failed, the script continued on and ran
>  > the tests as usual, which all pass since we were still running in the
>  > python2
>  > env, which was quite misleading. There may be a better place to stick the
>  > 'exit 1' or similar.
>  > 
>  
>  Hmm, might need to move the check elsewhere, since the run-tests script
>  is running in a sub-shell it should be file, but this will cause issues
>  if a we are activating the environment manually.
>  
>  Will submit a patch soon. :)
>  
>  _______________________________________________
>  python-bugzilla mailing list
>  python-bugzilla(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/python-bugzilla
>