Future directions for the Beaker test harness
by Nick Coghlan
A few weeks back, Amit put together a proof of concept for running the
test harness in a container, rather than directly on the host
(http://gerrit.beaker-project.org/#/c/3199).
That proof of concept relies on restraint, the new reference harness,
that is intended to eventually replace beah
(https://beaker-project.org/dev/proposals/reference-harness.html)
At the same time, I don't think restraint is currently getting the level
of review and testing that it needs to mature into a plausible
replacement for beah as the default harness.
I think Amit's proposed patch provides a possible way forward:
1. Accept the initial approach where restraint is the *only* supported
harness when running inside a container. Specifying both
"contained_harness" and "harness" as ks_meta variables should be an
error at this point (side note: 'harness' should also be documented
along with all the other ks_meta variables, with a link to
https://beaker-project.org/docs/alternative-harnesses/).
2. Recommend publishing both beah *and* restraint in the harness repos
for Beaker installations. This will not only make restraint available
for container based testing, but also make it readily available via
"harness=restraint" for normal testing, without needing to add a custom
repo definition.
3. Once we have container based testing working reliably with restraint,
drop the restriction against using alternative harnesses in containers.
The priority at the moment though is to get something working that can
run on an Atomic Host, and still provide a relatively normal execution
environment for the executed tasks. Supporting alternative harnesses
*inside* containers is a nice-to-have that can wait until later - by
flatly disallowing it, we ensure we don't have to spend any time working
on container related issues that don't impact restraint. For the initial
iteration, we can also ignore the question of choosing the base image
used to run the harness, as well as being able to start and stop other
containers on the host.
I've filed an RFE for 0.18 on that basis:
https://bugzilla.redhat.com/show_bug.cgi?id=1131388
As part of this, we may also want to move restraint from Bill's personal
account on GitHub to the main Beaker project account, but I don't think
that's particularly urgent at this point.
Regards,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
6 years, 1 month
Beaker inventory: smolt & lshw comparison
by Amit Saha
Hi all,
As part of the migration to "lshw" for beaker's inventory task, I ran a comparison of the "smolt"
based tasks with the "lshw" based task.
The "raw" data is available in the "comparison.html" file under each arch sub-directory at:
https://amitksaha.fedorapeople.org/beaker-inventory-comparison/comparison...
Here are the summarized differences:
** i686/AMD/athlon (32-bit system)
CPU flags
=========
* lshw categorises "fpu_exception" as a CPU flag, adds "wp" as cpu flag too.
(This is because we assume all the stuff lshw lists as CPU capabilities,
we consider them flags)
Others
======
* USBID retrieved by lshw with repetition, smolt didn't retrieve any
* Same PCIID retrieved by lshw, however there is repetition of a device (for PCIID as well)
* lshw is not able to detect the system model (same as smolt)
* lshw sets arch to i686 instead of i386 by smolt
* lshw fails to get the system vendor (same as smolt)
* lshw fails to get the FORMFACTOR
* CPUVENDOR obtained by smolt is AuthenticAMD, lshw sets it to "Advanced Micro Devices [AMD]"
** ia64
CPU flags
=========
* No CPU flags retrieved by lshw
Others
======
* Bulk of the USB IDs retrieved are 0000:0000s
* lshw is not able to detect the system model
* CPUFAMILY is reported by smolt as 1, by lshw as 2
* lshw fails to get the system vendor
* lshw fails to get the FORMFACTOR, same as smolt
* CPUVENDOR obtained by smolt is genuineintel, lshw sets it to "Intel corp."
* CPU stepping obtained by smolt is 0, whereas it is None by lshw
* PCIIDS retrieved by lshw contain one missing PCIID and a number of repetitions
* Memory reported by smolt is 7863 MB, 8192 by smolt. (**Needs more investigation**)
** ppc64/Power 6
CPU flags
=========
* No CPU flags retrieved by either
Others
======
* USBID retrieved by lshw, none by smolt, contains one repetition
* lshw is not able to detect the system model, same as smolt
* lshw fails to get the system vendor
* lshw reports a repeating PCIID (same as smolt)
** ppc64/Power 7
CPU flags
=========
* CPU flags retrieved by lshw as "performance-monitor"
Others
======
* No USB devices reported by either
* lshw is not able to detect the system model, same as smolt
* lshw fails to get the system vendor
* lshw and smolt both doesn't report any PCIIDs
** s390
CPU flags
=========
* CPU flags reported by lshw are each duplicated
Others
======
* No USBIDs retrieved by lshw, none by smolt
* lshw is not able to detect the system model, same as smolt
* lshw fails to get the system vendor, same as smolt
* lshw reports a repeating PCIID (same as smolt)
* Memory is reported as 2001 MB by both, system has 2048 MB
** x86_64/amd
CPU flags
=========
* lshw uses x86-64 in place of lm
* lshw categorises "fpu_exception" and "cpufreq" as a CPU flags
(This is because we assume all the stuff lshw lists as CPU capabilities,
we consider them flags)
Others
======
* More USBID retrieved by lshw, however there is repetition of a device (for PCIID as well)
* lshw is not able to detect the system model
* lshw sets VIRT_IOMMU to False, whereas smolt has it to True
* lshw sets arch to x86-64 instead of x86_64
* smolt gets CPUMODEL as " AMD Eng Sample, 1S256146U8K54_35/25/20_2/8 ", lshw gets it as "Opteron"
* lshw fails to get the system vendor
* lshw fails to get the FORMFACTOR
* CPUVENDOR obtained by smolt is AuthenticAMD, lshw sets it to "Advanced Micro Devices [AMD]"
* smolt gets 3811 MB of memory, lshw gets None on an 3919 MB system (**Needs more investigation**)
** x86_64/intel
CPU flags
=========
* lshw uses x86-64 in place of lm
* lshw categorises "fpu_exception" and "cpufreq" as a CPU flags
(This is because we assume all the stuff lshw lists as CPU capabilities,
we consider them flags)
Others
======
* More USBID retrieved by lshw, however there is repetition of a device
* lshw is not able to detect the system model
* lshw sets arch to x86-64 instead of x86_64
* smolt gets CPUMODEL as "Intel(R) Xeon(TM) CPU 3.60GHz", lshw gets it as "Xeon"
* lshw fails to get the system vendor
* CPUVENDOR/vendor obtained by smolt is GenuineIntel, lshw sets it to Intel Corp.
* lshw gets 4096 MB of memory correctly
** x86_64/kvm guest
CPU flags
=========
* lshw uses x86-64 in place of lm
* lshw categorises "fpu_exception" as a CPU flag
(This is because we assume all the stuff lshw lists as CPU capabilities,
we consider them flags)
Others
======
* USBID retrieved by lshw, none by smolt
* lshw is not able to detect the system model
* lshw sets arch to x86-64 instead of x86_64
* lshw fails to get the system vendor
* CPUVENDOR/vendor obtained by smolt is GenuineIntel, lshw sets it to Intel Corp.
* lshw fails finds one device less (PCIID)
* lshw gets 2048 MB of memory correctly
** Common to all archs
For all the architectures, the devices list returned by the lshw task is fewer than those
returned by smolt. The devices need to be individually compared to come up with a more accurate
answer.
PS: arm 32-bit and 64-bit comparison: Not possible, since we cannot run any smolt supported distro
on these.
The next plan of action is to fix the issues that we have found above - either in "beaker-system-scan"
or "lshw" itself.
Best,
Amit.
--
Amit Saha
SED, Hosted & Shared Services
Red Hat, Inc.
9 years
Running Beaker's tests in a Docker container
by Amit Saha
While working with beaker‘s code base, I often feel the need to run my tests for a patch/feature
and continue to work on with different things while they run, including running other tests
testing something different. Currently this is not possible since we start off with a clean
database on every test run and simultaneous runs would obviously make one run step on another’s feet.
I finally have an initial docker based prototype for making this possible:
https://github.com/amitsaha/docker_files/tree/master/dev_workflow/beaker-...
Best,
Amit.
9 years, 1 month
Integration Tests results
by Jaroslav Kortus
Hi there,
I've just executed full integration tests run with the following result (patched develop
branch):
Ran 1345 tests in 6731.735s
FAILED (SKIP=3, errors=70, failures=55)
real 112m14.756s
user 101m11.856s
sys 7m42.248s
Quite some errors then. Much more than I expected :). Looking at the results I have mixed
feelings.
Some selenium tests were failing (almost all in my case). When I ran the same test alone
it passed with no problems.
To get more data I ran the suite with "-sv" switch. Looking at the output there are some
things that I'm not sure are errors or not.
For instance:
bkr.server.mail: ERROR: Exception thrown when trying to send mail
[...]
error: [Errno 111] Connection refused
bkr.server.tests.data_setup: DEBUG: Marked R:706 as complete with result Pass
or:
FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line
55, in test_all_tables_use_innodb
table), 'InnoDB')
AssertionError: u'MyISAM' != 'InnoDB'
I've not created any new tables there (only added columns to existing tables). So how can
this happen?
So the main question here is, are all the tests expected to pass on the develop branch? Or
what are the usual results I can compare my patches to? Are there any other requirements
besides the deps of integration tests RPM and
https://beaker-project.org/dev/guide/writing-a-patch.html#testing-your-pa...
Thank you,
Jaroslav.
PS: are the results saved somewhere or is stdout the only option? "--debug-log" looks more
like the framework debugging than the actual tests' debug output...
9 years, 2 months
[PATCH] bkr-client-autocompletion: grep -> /usr/bin/grep
by Andrej Manduch
When you have strange things in your ~/.basrc (like grep --color=always)
then beaker autocompletion can produce wrong output.
example:
burlak@borg /e/bash_completion.d $ bkr
-h
^[[m^[[Kpolicy-revoke^[[01;31m^[[K
--help
^[[m^[[Kremove-account^[[01;31m^[[K
^[[m^[[Kdistros-edit-version^[[01;31m^[[K
^[[m^[[Ksystem-delete^[[01;31m^[[K
^[[m^[[Kdistros-list^[[01;31m^[[K
^[[m^[[Ksystem-details^[[01;31m^[[K
^[[m^[[Kdistros-tag^[[01;31m^[[K
This patch will ensure that bkr autocompletion will
always use original `grep`.
Signed-off-by: Andrej Manduch <amanduch(a)gmail.com>
---
Client/bash-completion/bkr | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/Client/bash-completion/bkr b/Client/bash-completion/bkr
index 76d6b9e..f64e1c0 100644
--- a/Client/bash-completion/bkr
+++ b/Client/bash-completion/bkr
@@ -7,9 +7,10 @@
# Function expands options from main option (like workflow-tcms)
_bkr_complete()
{
+ local GREP=`which grep --skip-alias`
local component=$( echo $COMP_LINE | cut -d' ' -f2 )
if [ $component != "" ] ; then
- local opts=$(echo $( bkr $component --help 2>/dev/null | grep -o -- '--[^ =]*' | sort | uniq ) --help )
+ local opts=$(echo $( bkr $component --help 2>/dev/null | $GREP -o -- '--[^ =]*' | sort | uniq ) --help )
[[ $COMP_LINE == bkr\ $component* ]] && \
printf %s "$opts"
@@ -20,6 +21,7 @@ _bkr_complete()
_bkr()
{
COMPREPLY=()
+ local GREP=`which grep --skip-alias`
local bkr=$1
local cur prev
local -a words
@@ -30,7 +32,7 @@ _bkr()
fi
# Commands offered as completions for bkr
- local cmds=( $( bkr --help 2>/dev/null | grep " " | cut -d' ' -f3 | sed '1d' | cut -d'=' -f1 ) -h --help )
+ local cmds=( $( bkr --help 2>/dev/null | $GREP " " | cut -d' ' -f3 | sed '1d' | cut -d'=' -f1 ) -h --help )
local i c cmd
for (( i=1; i < ${#words[@]}-1; i++ )) ; do
--
2.0.0.GIT
9 years, 2 months
Fedora COPR now supports package signing
by Nick Coghlan
We've previously discussed the idea of allowing Beaker to depend on EPEL
when running on RHEL/CentOS, thus allowing the use of Fedora's COPR
system for package builds.
The COPR team recently switched COPR over to signing packages by
default, so it may be worth revisiting that idea at some point in the
coming weeks/months.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
9 years, 2 months
[PATCH] bkr-client-autocompletion: grep -> /usr/bin/grep
by Andrej Manduch
From: Andrej Manduch <amanduch(a)gmail.com>
When you have strange things in your ~/.basrc (like grep --color=always)
then beaker autocompletion can produce wrong output.
example:
burlak@borg /e/bash_completion.d $ bkr
-h
^[[m^[[Kpolicy-revoke^[[01;31m^[[K
--help
^[[m^[[Kremove-account^[[01;31m^[[K
^[[m^[[Kdistros-edit-version^[[01;31m^[[K
^[[m^[[Ksystem-delete^[[01;31m^[[K
^[[m^[[Kdistros-list^[[01;31m^[[K
^[[m^[[Ksystem-details^[[01;31m^[[K
^[[m^[[Kdistros-tag^[[01;31m^[[K
This patch will ensure that bkr autocompletion will
always use original `grep`.
Signed-off-by: Andrej Manduch <amanduch(a)gmail.com>
---
Client/bash-completion/bkr | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/Client/bash-completion/bkr b/Client/bash-completion/bkr
index 76d6b9e..f64e1c0 100644
--- a/Client/bash-completion/bkr
+++ b/Client/bash-completion/bkr
@@ -7,9 +7,10 @@
# Function expands options from main option (like workflow-tcms)
_bkr_complete()
{
+ local GREP=`which grep --skip-alias`
local component=$( echo $COMP_LINE | cut -d' ' -f2 )
if [ $component != "" ] ; then
- local opts=$(echo $( bkr $component --help 2>/dev/null | grep -o -- '--[^ =]*' | sort | uniq ) --help )
+ local opts=$(echo $( bkr $component --help 2>/dev/null | $GREP -o -- '--[^ =]*' | sort | uniq ) --help )
[[ $COMP_LINE == bkr\ $component* ]] && \
printf %s "$opts"
@@ -20,6 +21,7 @@ _bkr_complete()
_bkr()
{
COMPREPLY=()
+ local GREP=`which grep --skip-alias`
local bkr=$1
local cur prev
local -a words
@@ -30,7 +32,7 @@ _bkr()
fi
# Commands offered as completions for bkr
- local cmds=( $( bkr --help 2>/dev/null | grep " " | cut -d' ' -f3 | sed '1d' | cut -d'=' -f1 ) -h --help )
+ local cmds=( $( bkr --help 2>/dev/null | $GREP " " | cut -d' ' -f3 | sed '1d' | cut -d'=' -f1 ) -h --help )
local i c cmd
for (( i=1; i < ${#words[@]}-1; i++ )) ; do
--
2.1.0
9 years, 2 months
Receiving HTML rather than JSON despite Accept header
by Nick Coghlan
Hi folks,
I'm trying to use the HTTP APIs documented in https://beaker-project.org/docs-release-19/server-api/http.html as follows:
curl --negotiate -H 'Accept: application/json' https://<Beaker server domain>
I have a valid Kerberos ticket, but the main page just returns the HTML list of systems, while the subcollections for available, free and my systems return a login page redirect.
So it appears either something is broken or the API documentation has omitted a required setup step.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
9 years, 2 months
Building beaker from sources
by Jaroslav Kortus
Hi there,
I've recently come to a need of building beaker from git.
Unfortunately there's couple of obstacles I'd like your advice on.
First: no tito anymore. Last time I was doing the rebuilds from upstream devel branch it
was using tito which was quite handy for generating srpms which I then had rebuilt. Is
there anything helpful in the same way today?
I found it quite difficult compared to my previous experience. Even the deps listed in the
spec file are hard to download (wget downloads it as a different name from what's expected
in the spec file).
Second: beaker-transfer does not start "green" any more without archive server config
options present. So the question is: do I need that daemon for anything really?
I've ran the jobs in my current combo setup (LC+server on one machine) and it seemed to me
to transfer the logs just as before with the daemon (beaker-transfer) down.
Third: I would like to have some step-by-step howto on building the packages from sources.
I could not find it on the website. Something similar to kickstarts for custom-deployment
of beaker. With all necessary repos linked as well as commands that you use to build the
final packages.
Thanks for any pointers that could resolve my issues :).
Cheers,
Jaroslav.
9 years, 2 months