Implicit retries considered useful?
by Nick Coghlan
To help narrow down a problem that sometimes occurs with lab controller
<-> main server communications, Beaker 0.13.2 is currently scheduled to
completely remove the implicit retries for the affected XML-RPC calls
(see https://bugzilla.redhat.com/show_bug.cgi?id=974352).
However, it occurs to me that there's a potentially big downside to
doing this: currently, we can usually restart the main web service (e.g.
for a maintenance update that doesn't include any database changes)
without significant ill effects, because it will just trip the automatic
retries in the lab controllers for any in progress calls and those calls
will generally succeed on the second attempt.
By turning the implicit retries off *completely* for the lab controller
daemons (rather then just reducing the number of attempts and logging
each failed attempt properly), any calls back to the server when the
server shuts down will actually *fail* completely.
So while our initial assessment was that the implicit retries were doing
more harm than good, I think this may be a case where we actually need
them. If I'm right, then 0.13.2 isn't shippable in its current form (we
need to either revert that change to the lab controllers, or update it
to still retry at least once, and log the first failure properly).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 9 months
Initial coverage results for the Beaker integration test suite
by Nick Coghlan
I ran the Beaker integration test suite as:
./run-tests.sh --with-progressive --with-coverage
--cover-package=bkr --cover-erase --cover-tests --cover-html
--cover-inclusive
I also added a "sitecustomize.py" to the integration source directory
that enables coverage data collection as described in
http://nedbatchelder.com/code/coverage/subprocess.html
However, I forgot to set COVERAGE_PROCESS_START when running the tests,
so I suspect it didn't trigger properly (hence the lack of coverage for
the client commands).
Full results are included below (unfortunately, the HTML report
generation failed, or I would post those somewhere)
Overall reported coverage is 71%, but there are a few caveats on that:
* As noted above, I'm pretty sure I didn't enable subprocess coverage
properly, which deflates the numbers.
* I included the tests in the coverage data to check they were being run
currently. I don't have krb or qpid testing set up, so the gaps in the
test coverage numbers seem reasonable to me. Since the coverage for
these is generally pretty high, it inflates the overall reported coverage.
* The "--cover-inclusive" option is designed to check for source files
which aren't imported at all. However, because it's based on the working
directory, it doesn't work properly for us - running from
IntegrationTests means it can't spot missed files from Common,
LabController, Client or Server.
Ned favours using the coverage CLI to invoke nose tests over relying on
the Nose coverage plugin. There's also the nose-cov plugin which is
designed to work with a .coveragerc file (unlike the default plugin
provided by nose). I'll look into one or both of those to try to get the
"not even imported" data directly as part of the coverage execution (as
well as to hopefully get working HTML output).
Still, these are very promising numbers for a first analysis :)
Cheers,
Nick.
Name Stmts
Miss Cover Missing
------------------------------------------------------------------------------------------
bkr 5
5 0% 2-6
bkr.client 478
267 44% 16-20, 32-34, 46-55, 60-61, 75-316, 321-330, 334-337,
341-345, 349-357, 362-415, 437, 448-451, 453, 455, 459, 476, 481-497,
501-509, 513-523, 527-528, 532-540, 582-591, 593-596, 598-602, 604, 607,
609, 611-614, 616-619, 627-632, 634-638, 641, 643, 646, 682, 684-687,
698, 702, 710-712, 715-717, 720-722, 728, 733, 736, 741, 744, 749, 752,
769, 784-791, 795-796, 799, 802, 807, 810
bkr.common 1
0 100%
bkr.common.bexceptions 7
1 86% 7
bkr.common.helpers 142
90 37% 15-16, 25-36, 40, 44-48, 54-60, 63-64, 67-82, 86-88, 99-101,
105-113, 117-125, 129-137, 140, 143-146, 160, 163-167, 177, 191-193
bkr.common.message_bus 111
75 32% 15-17, 41, 50-60, 69-88, 91-100, 103-105, 108-110, 113-134,
138-150, 154-159, 162
bkr.inttest 148
9 94% 150, 154, 156, 185, 187, 276-279
bkr.inttest.assertions 42
12 71% 30, 35-36, 40, 53, 55, 72, 76-79, 88
bkr.inttest.client 41
0 100%
bkr.inttest.client.test_common_options 7
0 100%
bkr.inttest.client.test_distro_trees_list 69
1 99% 39
bkr.inttest.client.test_distro_trees_verify 17
0 100%
bkr.inttest.client.test_distros_edit_version 13
0 100%
bkr.inttest.client.test_distros_list 29
1 97% 34
bkr.inttest.client.test_distros_tag 12
0 100%
bkr.inttest.client.test_distros_untag 12
0 100%
bkr.inttest.client.test_group_create 77
5 94% 46, 76, 88, 98, 119
bkr.inttest.client.test_group_members 24
1 96% 45
bkr.inttest.client.test_group_modify 302
19 94% 60, 73, 85, 98, 174, 209, 226, 272, 282, 301, 341, 350, 360,
386, 395, 433, 442, 485, 494
bkr.inttest.client.test_job_cancel 42
3 93% 32, 42, 54
bkr.inttest.client.test_job_clone 31
1 97% 40
bkr.inttest.client.test_job_delete 63
2 97% 44, 83
bkr.inttest.client.test_job_list 38
0 100%
bkr.inttest.client.test_job_logs 54
1 98% 66
bkr.inttest.client.test_job_modify 80
9 89% 60-67, 96
bkr.inttest.client.test_job_results 25
1 96% 35
bkr.inttest.client.test_job_submit 27
0 100%
bkr.inttest.client.test_job_watch 25
1 96% 28
bkr.inttest.client.test_list_labcontrollers 10
0 100%
bkr.inttest.client.test_list_systems 95
3 97% 133, 181, 193
bkr.inttest.client.test_machine_test 43
0 100%
bkr.inttest.client.test_system_delete 35
2 94% 28, 53
bkr.inttest.client.test_system_details 13
0 100%
bkr.inttest.client.test_task_add 12
1 92% 12
bkr.inttest.client.test_task_details 82
0 100%
bkr.inttest.client.test_task_list 45
0 100%
bkr.inttest.client.test_update_prefs 25
2 92% 26, 33
bkr.inttest.client.test_watchdog_extend 12
0 100%
bkr.inttest.client.test_watchdogs_extend 14
0 100%
bkr.inttest.client.test_workflow 29
0 100%
bkr.inttest.client.test_workflow_simple 77
1 99% 51
bkr.inttest.data_setup 431
14 97% 79-81, 169, 218, 577-585
bkr.inttest.labcontroller 57
10 82% 23, 73-79, 90-93
bkr.inttest.labcontroller.test_distro_import 163
1 99% 17
bkr.inttest.labcontroller.test_logs 38
4 89% 30-33
bkr.inttest.labcontroller.test_provision 30
0 100%
bkr.inttest.labcontroller.test_proxy 582
18 97% 321-326, 329-334, 412, 428, 444, 520, 598, 681
bkr.inttest.labcontroller.test_watchdog 46
0 100%
bkr.inttest.mail_capture 34
3 91% 63-67
bkr.inttest.message_bus 7
3 57% 9-11
bkr.inttest.server 0
0 100%
bkr.inttest.server.selenium 112
7 94% 80, 101, 147, 160-163
bkr.inttest.server.selenium.test_activity 89
0 100%
bkr.inttest.server.selenium.test_add_group 29
1 97% 36
bkr.inttest.server.selenium.test_add_system 114
3 97% 44, 162, 192
bkr.inttest.server.selenium.test_add_user 156
1 99% 228
bkr.inttest.server.selenium.test_cancel 51
0 100%
bkr.inttest.server.selenium.test_csv_export 103
0 100%
bkr.inttest.server.selenium.test_csv_import 76
1 99% 113
bkr.inttest.server.selenium.test_csv_roundtrip 42
1 98% 54
bkr.inttest.server.selenium.test_distro_search 116
0 100%
bkr.inttest.server.selenium.test_distro_trees 198
0 100%
bkr.inttest.server.selenium.test_distros 155
4 97% 61, 74, 167, 175
bkr.inttest.server.selenium.test_external_reports 39
0 100%
bkr.inttest.server.selenium.test_group_edit 431
0 100%
bkr.inttest.server.selenium.test_groups 69
1 99% 24
bkr.inttest.server.selenium.test_item_count 46
1 98% 53
bkr.inttest.server.selenium.test_job_ack 40
0 100%
bkr.inttest.server.selenium.test_job_delete 91
1 99% 127
bkr.inttest.server.selenium.test_job_delete_xmlrpc 44
8 82% 32, 34, 39, 41, 46, 48, 53, 55
bkr.inttest.server.selenium.test_job_export_xml 32
0 100%
bkr.inttest.server.selenium.test_job_matrix 213
0 100%
bkr.inttest.server.selenium.test_job_search 130
1 99% 27
bkr.inttest.server.selenium.test_jobs 493
2 99% 167, 293
bkr.inttest.server.selenium.test_jobs_xmlrpc 47
3 94% 36, 93, 122
bkr.inttest.server.selenium.test_labcontrollers 306
0 100%
bkr.inttest.server.selenium.test_log_upload 31
0 100%
bkr.inttest.server.selenium.test_login 127
20 84% 30-31, 102, 121, 141, 148-164, 191
bkr.inttest.server.selenium.test_manual_system_notloggedin 18
0 100%
bkr.inttest.server.selenium.test_motd 21
0 100%
bkr.inttest.server.selenium.test_my_menu_tests 30
3 90% 27-28, 35
bkr.inttest.server.selenium.test_osversions 77
0 100%
bkr.inttest.server.selenium.test_prefs 105
1 99% 136
bkr.inttest.server.selenium.test_product 23
0 100%
bkr.inttest.server.selenium.test_provision 85
1 99% 116
bkr.inttest.server.selenium.test_recipe_search 37
0 100%
bkr.inttest.server.selenium.test_recipes 155
11 93% 189-204
bkr.inttest.server.selenium.test_recipes_xmlrpc 75
0 100%
bkr.inttest.server.selenium.test_recipetasks_xmlrpc 53
0 100%
bkr.inttest.server.selenium.test_remove_lc 62
1 98% 19
bkr.inttest.server.selenium.test_reserve_system 167
1 99% 218
bkr.inttest.server.selenium.test_retentiontag 66
1 98% 43
bkr.inttest.server.selenium.test_system_actions 133
0 100%
bkr.inttest.server.selenium.test_system_available 205
0 100%
bkr.inttest.server.selenium.test_system_groups 94
0 100%
bkr.inttest.server.selenium.test_system_loan 141
0 100%
bkr.inttest.server.selenium.test_system_menu_tests 47
1 98% 56
bkr.inttest.server.selenium.test_system_note 105
4 96% 121, 132, 147, 158
bkr.inttest.server.selenium.test_system_return 64
0 100%
bkr.inttest.server.selenium.test_system_search 527
22 96% 204, 206, 208, 217, 219, 228, 230, 243, 245, 247, 256, 258,
260, 269, 271, 273, 282, 284, 286, 299, 301, 303
bkr.inttest.server.selenium.test_system_take 222
19 91% 80, 94, 107, 124, 139, 151, 165, 177, 196, 214, 238,
256-257, 269, 275-278, 286
bkr.inttest.server.selenium.test_system_view 585
0 100%
bkr.inttest.server.selenium.test_systems 157
1 99% 111
bkr.inttest.server.selenium.test_systems_xmlrpc 598
13 98% 46, 59, 73, 114, 155, 172, 216, 300, 314, 391, 472, 485, 514
bkr.inttest.server.selenium.test_task_by_name 20
0 100%
bkr.inttest.server.selenium.test_task_search 141
1 99% 131
bkr.inttest.server.selenium.test_taskactions_xmlrpc 17
0 100%
bkr.inttest.server.selenium.test_tasks 134
1 99% 196
bkr.inttest.server.selenium.test_utilisation_graph 12
0 100%
bkr.inttest.server.selenium.test_watchdogs 27
0 100%
bkr.inttest.server.test_ack_in_jobxml 27
0 100%
bkr.inttest.server.test_beakerd 802
5 99% 212-214, 748, 1185
bkr.inttest.server.test_deadrecipes 53
0 100%
bkr.inttest.server.test_group_xml 45
0 100%
bkr.inttest.server.test_jobs 40
1 98% 23
bkr.inttest.server.test_kickstart 388
8 98% 34-43, 1445
bkr.inttest.server.test_lab_controller 20
0 100%
bkr.inttest.server.test_mail 125
1 99% 188
bkr.inttest.server.test_model 1230
16 99% 31, 726, 825, 836, 1039-1058, 1675, 2243
bkr.inttest.server.test_rdf 30
0 100%
bkr.inttest.server.test_reporting_queries 275
0 100%
bkr.inttest.server.test_system_status 44
0 100%
bkr.inttest.server.test_tasks 29
0 100%
bkr.inttest.server.test_update_status 276
2 99% 28, 307
bkr.inttest.server.test_utilisation 84
0 100%
bkr.inttest.server.tools 0
0 100%
bkr.inttest.server.tools.test_log_delete 160
9 94% 15-16, 20-21, 45-49, 111
bkr.inttest.server.tools.test_nag_mail 54
0 100%
bkr.inttest.server.tools.test_refresh_ldap 19
0 100%
bkr.inttest.server.webdriver_utils 52
6 88% 14-15, 42, 48, 50, 52
bkr.labcontroller 0
0 100%
bkr.labcontroller.config 20
1 95% 32
bkr.labcontroller.log_storage 76
55 28% 11-13, 16, 22, 25-52, 55-56, 59, 62-68, 88-95, 102-109, 116-123
bkr.labcontroller.proxy 484
368 24% 36-37, 42, 52, 57, 63, 93-97, 106-107, 116-117, 127-128,
138-139, 149-150, 157-159, 162-163, 168-169, 174, 183-200, 206-209,
215-279, 282-289, 297-304, 309-346, 351-353, 358-360, 365-366, 369-372,
377-383, 388-400, 403-409, 413, 418-443, 455-460, 467, 490-494, 501-502,
509-511, 515-518, 523-524, 529-530, 535-536, 539-540, 545-546, 555-556,
578-582, 587, 592, 603, 609, 615, 621, 629, 634-635, 638-641, 651-668,
672-679, 682-702, 705-712, 718-744, 747-754, 758-762, 765-769, 772-776,
781-786, 789-794, 797-806, 809-819, 822-832, 835-843
bkr.labcontroller.utils 51
29 43% 15-17, 19, 34-44, 47-48, 51-52, 74-84
bkr.log 67
40 40% 9-13, 50-53, 61-63, 74-80, 85-107, 118-122, 126, 130
bkr.server 3
0 100%
bkr.server.BasicAuthTransport 24
19 21% 8-10, 15-47
bkr.server.CSV_import_export 603
516 14% 15-22, 57, 67, 77-81, 91-157, 161-166, 173-180, 187-196,
203-216, 219-225, 240-241, 248-358, 361-378, 392-424, 428-430, 433-439,
447-449, 452-459, 463-470, 479-484, 492-541, 544-552, 560-562, 569-661,
664-666, 669-693, 701-706, 713-761, 764-767, 775-777, 784-806, 809-811,
819-821, 828-852, 855-857
bkr.server.activity 85
53 38% 27-43, 46-49, 54-60, 66-72, 79-85, 91-97, 103-109, 115-121,
126-151
bkr.server.admin_page 42
15 64% 15, 20, 23, 28, 49, 52-64
bkr.server.authentication 82
49 40% 24-35, 58-62, 69-71, 85-93, 113-141, 151-152
bkr.server.bexceptions 11
0 100%
bkr.server.breadcrumbs 16
10 38% 33-44, 56
bkr.server.commands 19
7 63% 40-51
bkr.server.configuration 93
57 39% 54-76, 91-112, 118-131, 142-147, 152-160, 165-171
bkr.server.controller_utilities 229
163 29% 23-29, 32-34, 38-44, 52-79, 86-88, 100-114, 127-128,
136-154, 158, 162, 166, 170, 174, 178, 182, 186, 190, 194, 198, 202,
206, 210, 214, 218, 222, 226, 230, 234, 238, 242-247, 251, 255, 260,
265, 269, 273-278, 283-346, 351-362
bkr.server.controllers 1314
1018 23% 93-96, 102-105, 112-118, 127-143, 183-186, 234-252,
257-259, 263, 267, 271, 275, 280, 284, 288-306, 310, 314-322, 326-335,
339-362, 369, 379, 388, 397, 406-437, 440-444, 447-458, 463-479,
484-584, 603-606, 612-643, 649-671, 677-705, 710-712, 721-745, 750-831,
888-900, 905-918, 926-933, 937-949, 954-992, 997-1006, 1017-1031,
1038-1053, 1058-1083, 1104-1112, 1125-1143, 1150-1175, 1186-1279,
1286-1399, 1404-1426, 1432-1448, 1453-1474, 1479-1487, 1492-1533,
1544-1593, 1598-1610, 1615-1661, 1667-1730, 1734, 1739-1800, 1806,
1810-1819, 1823-1841, 1846-1854, 1858-1861, 1866-1881, 1887-1888, 1892,
1896-1898
bkr.server.distro 202
146 28% 39-46, 62-68, 74-78, 86-98, 103-115, 120-121, 127-141,
144-159, 162-166, 171-173, 178-181, 184-221, 265-285, 304-336, 354-364,
372-382
bkr.server.distro_family 10
4 60% 11-18
bkr.server.distrotrees 158
127 20% 29-73, 83-89, 93-108, 122-148, 153-177, 184-197, 202-229,
276-322
bkr.server.enum 69
5 93% 27, 65-66, 77, 107
bkr.server.external_reports 45
20 56% 32-35, 43, 54, 63-67, 74-83
bkr.server.group 545
426 22% 32-33, 36-38, 62-65, 129-136, 141-159, 163-165, 170,
180-199, 204-214, 219-250, 270-298, 308-311, 320-358, 364-381, 386-419,
425-446, 451-454, 460-465, 468-502, 508-539, 558, 565-590, 608, 614-637,
643-659, 665-681, 685-693, 697-705, 728-760, 791-875, 892-908
bkr.server.helpers 27
20 26% 7-11, 15, 20, 25-37
bkr.server.identity 91
56 38% 49-64, 67-79, 87-114, 117-129
bkr.server.installopts 56
8 86% 18, 21-24, 39-40, 70, 80
bkr.server.job_matrix 191
160 16% 16-17, 20-23, 26-36, 48-83, 88-90, 98-115, 125-133,
143-156, 164-167, 175-305, 313, 324-340
bkr.server.job_utilities 62
47 24% 35-59, 63-80, 89-102, 110-125
bkr.server.jobs 556
337 39% 66-69, 113-114, 119, 128-131, 134-137, 143-149, 155-161,
183-188, 226-278, 299-324, 340-347, 357-413, 431-432, 453, 463-464, 466,
469, 478-479, 486, 492-493, 495, 507-508, 521-547, 550-554, 573-574,
591-592, 594-595, 616-618, 621, 626-632, 637-676, 682-686, 690-697,
701-707, 715-723, 727-728, 734, 740, 744-803, 820-838, 846-854, 868-899,
903-943
bkr.server.jobxml 230
85 63% 30-39, 45, 48-52, 55-59, 63-65, 71-80, 104-105, 116-117,
124, 128, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 198, 218,
224, 228, 233-235, 249, 255-259, 263-265, 273, 277-281, 285-287, 298-321
bkr.server.keytypes 63
30 52% 47, 56-65, 75-83, 88-98, 108-111, 115-121
bkr.server.kickstart 107
9 92% 83, 100, 230-236
bkr.server.labcontroller 327
270 17% 35, 45-52, 64-109, 114-213, 218-222, 227-300, 309-339,
350-358, 363-372, 379-388, 393-405, 426-445, 456-466, 469-476, 482-490,
500-511, 515-520, 525-558
bkr.server.mail 88
36 59% 38, 65-66, 75, 78-97, 101-130, 137-138, 143, 169-170
bkr.server.message_bus 32
12 63% 16, 25-31, 34-39
bkr.server.metrics 36
18 50% 17-19, 22-26, 31-35, 40-41, 46-49
bkr.server.model 3745
1253 67% 93, 232, 235, 239, 267, 1540, 1572, 1577-1579, 1588, 1610,
1613, 1624-1641, 1654, 1695, 1716-1717, 1736, 1741, 1751-1753,
1757-1771, 1774-1787, 1791-1794, 1823, 1840-1844, 1868-1872, 1883, 1888,
1891-1900, 1907-1915, 1946, 2026-2040, 2056-2059, 2071, 2102, 2138,
2155, 2163, 2167-2174, 2178-2181, 2184-2191, 2197-2203, 2215-2218,
2236-2239, 2243-2251, 2256-2258, 2261, 2264, 2268, 2286-2296, 2299-2302,
2317-2357, 2360-2364, 2371, 2374-2377, 2384-2401, 2407-2418, 2426, 2434,
2439, 2449, 2462-2465, 2471-2530, 2537-2565, 2568-2581, 2584-2591,
2598-2615, 2622-2645, 2653-2671, 2678-2682, 2692-2695, 2701-2704,
2710-2727, 2733-2759, 2781, 2794, 2821, 2827, 2844, 2846, 2849, 2882,
2884, 2887, 2895, 2909, 2913, 2915, 2924, 2945, 2991, 2998, 3002, 3012,
3019, 3023, 3040, 3044, 3056, 3086, 3094, 3099-3101, 3111-3113,
3123-3125, 3129-3132, 3138, 3165, 3171, 3175, 3179-3180, 3184-3188,
3207, 3218-3221, 3249, 3268-3271, 3279, 3318, 3321, 3344, 3351, 3356,
3371-3372, 3386-3387, 3395, 3399-3403, 3411-3412, 3417-3418, 3423-3424,
3451-3454, 3463, 3473, 3477-3478, 3495-3498, 3546, 3558, 3585, 3628,
3664, 3678, 3682-3684, 3718, 3721, 3726, 3731, 3735, 3739, 3743, 3747,
3756, 3759-3767, 3770-3773, 3776-3779, 3784-3786, 3790, 3807-3808,
3822-3826, 3830, 3838, 3853, 3857, 3873, 3877, 3925, 3930, 3958, 3961,
3966-3968, 3976, 3987, 3997, 4030-4031, 4036-4037, 4052, 4095-4128,
4164-4168, 4172-4181, 4188-4205, 4211, 4231-4232, 4262-4265, 4270, 4272,
4280-4286, 4290-4296, 4304-4309, 4317-4318, 4322-4339, 4344-4396, 4416,
4421-4428, 4431, 4434-4437, 4440-4447, 4449-4454, 4459-4464, 4476-4488,
4505, 4511, 4514-4515, 4518, 4522, 4524, 4537, 4542, 4547, 4550-4552,
4555-4568, 4571-4584, 4638, 4671, 4705, 4709-4712, 4716-4719, 4723,
4727, 4731, 4735, 4739, 4743, 4750-4756, 4765-4767, 4786, 4800, 4804,
4812, 4826, 4829-4833, 4839, 4860-4864, 4877, 4884, 4905, 4909-4920,
4943, 4946-4949, 4952-4954, 4958, 4962, 4966, 4972, 4975, 4994-4996,
5012, 5021-5028, 5032-5034, 5042-5043, 5067, 5124, 5137-5142, 5148,
5153, 5176, 5179-5181, 5187, 5192, 5197-5199, 5221, 5280-5283, 5285,
5308-5309, 5325, 5387-5388, 5412-5413, 5427-5430, 5483-5486, 5528, 5610,
5625-5626, 5629, 5710, 5728-5732, 5738-5742, 5749-5752, 5759, 5766-5770,
5777, 5803, 5828-5831, 5896, 5960, 5993, 6012, 6018, 6026, 6033, 6038,
6061, 6063, 6069, 6081, 6087, 6094, 6096, 6104, 6145, 6151, 6157, 6160,
6167, 6183, 6196, 6215, 6233, 6243-6244, 6247-6250, 6264-6267, 6286,
6361, 6370, 6390-6401, 6411, 6466, 6470, 6505, 6521, 6532-6533, 6545,
6550, 6566, 6573, 6613, 6616, 6632, 6653, 6673-6709, 6713-6721,
6730-6746, 6767, 6772-6774, 6778-6780, 6788, 6798-6873, 6877, 6935-6940,
6942-6947, 6949-6954, 6956-6961, 6963-6968, 6970-6975, 6983-7008,
7027-7031, 7041-7044, 7053, 7062, 7065, 7068-7070, 7101, 7107-7113,
7123-7127, 7149, 7154-7157, 7159-7160, 7170, 7179, 7184, 7187-7191,
7194-7195, 7198-7228, 7232-7276, 7279-7293, 7296-7307, 7321, 7790-7793,
7797-7798, 7801-7809
bkr.server.needpropertyxml 659
76 88% 106, 114, 121, 124-128, 144, 165-172, 195, 199-206, 226,
230-237, 240, 339-340, 369, 425, 439, 471, 614, 741-747, 878-879,
888-893, 934, 948-952, 961, 971, 981, 999, 1107-1116
bkr.server.osversion 123
70 43% 61-66, 76-81, 91-101, 106-119, 125-136, 141-149, 154-164,
168-180
bkr.server.power 84
37 56% 41-43, 47, 52, 56, 85, 94-102, 112-121, 125-131, 136-148,
158-161
bkr.server.preferences 96
57 41% 50-59, 77-114, 127-133, 138-153, 160-164
bkr.server.rdf 118
54 54% 49-51, 54-58, 70-75, 86-103, 106-111, 114-119, 129, 132,
134, 136, 138, 144, 146, 149, 151, 153, 155, 157, 161, 163, 165, 167,
169, 171, 173-174
bkr.server.recipes 249
170 32% 82-88, 96-113, 127-131, 140-149, 158-166, 174-178, 186-194,
203-219, 227-232, 241-261, 268-288, 295-301, 304-313, 316-342, 347, 354,
358-408, 420-429, 434-442
bkr.server.recipesets 63
28 56% 62-70, 84-94, 102-110
bkr.server.recipetasks 127
89 30% 46-62, 70-85, 95-109, 118-122, 130-134, 142-146, 154-162,
170-178, 182-183, 188-210
bkr.server.reports 167
118 29% 37-40, 47, 53-55, 64-68, 73, 76-106, 116-131, 134-138,
142-145, 149-150, 163-224, 234-246, 249-269
bkr.server.reserve_workflow 96
69 28% 29-37, 43-78, 87-107, 115, 121-129, 133, 139-156
bkr.server.retention_tag_utility 15
11 27% 9-13, 17-22
bkr.server.retention_tags 98
54 45% 41, 52-59, 66-74, 78-81, 87-93, 98-104, 109-110, 121, 124-151
bkr.server.search_utility 855
543 36% 29, 39-57, 61, 65, 73, 77, 81, 111, 116, 119, 123, 129,
133, 137, 141, 147-149, 155-158, 165-171, 178-181, 188-194, 209-215,
253, 256, 259-260, 263-264, 267-272, 275-280, 283, 290-301, 312-316,
321-332, 337, 345, 350, 365-388, 391, 394-412, 415-418, 426-449, 461,
469-470, 486-487, 494-500, 504-507, 544-548, 561, 567, 587-593, 596,
599, 612-642, 645-653, 657-690, 693-702, 705, 709-722, 726, 730,
741-761, 765-785, 792-803, 811-820, 829-844, 850-855, 864-867, 872,
884-889, 891-893, 905-906, 958-961, 965-968, 972-975, 983-990,
1025-1061, 1066-1103, 1107-1122, 1126-1140, 1144-1148, 1152, 1156,
1160-1162, 1182-1189, 1331-1334, 1338-1340, 1344-1354, 1358, 1362, 1366,
1370, 1374, 1378-1381, 1385-1388, 1392-1394, 1398-1407, 1411-1429,
1485-1490, 1503-1509, 1525-1527
bkr.server.stdvars 10
5 50% 6-9, 12
bkr.server.system_action 30
16 47% 14-26, 32-38
bkr.server.systems 113
69 39% 27, 37-40, 55-59, 73-75, 90-97, 132-139, 149-152, 163-165,
204-241, 276-285, 302-310
bkr.server.tag 29
7 76% 33-36, 41-46
bkr.server.task_actions 46
25 46% 50, 61-67, 75, 93-112
bkr.server.tasks 252
190 25% 70-77, 115-175, 189-196, 204-221, 227-232, 238, 241-308,
315-324, 332-333, 338-371, 383-405, 413-414, 421, 428, 431-447, 450-454,
458-460
bkr.server.tools 0
0 100%
bkr.server.tools.beakerd 519
204 61% 69-87, 105-107, 111, 126, 133-135, 144-146, 177-179, 196,
203-205, 266-269, 309-313, 318, 336-338, 411-412, 421-423, 426-428,
432-436, 485-486, 526-579, 582-606, 618, 626-628, 704-715, 721, 734-738,
743-779, 783, 786-788, 791, 795-802, 806-812, 816-855, 858
bkr.server.tools.init 166
20 88% 44, 67, 72-75, 220-237, 240-244, 249
bkr.server.tools.log_delete 67
19 72% 29-50, 71, 86-91, 95
bkr.server.tools.nag_email 58
26 55% 30-39, 42-43, 46-59, 77-78, 85
bkr.server.tools.refresh_ldap 29
12 59% 28-46, 49
bkr.server.user 145
85 41% 65, 75-83, 90-105, 108-114, 119-133, 144-158, 163-173,
178-182, 187-194, 198-219, 222-223, 227-228, 231-232
bkr.server.util 94
39 59% 39-57, 79-82, 91, 135-148, 158-162, 175
bkr.server.utilisation 40
1 98% 20
bkr.server.validators 128
94 27% 11-15, 26, 29-43, 60-74, 84-116, 128-134, 139-143, 149-153,
157-165, 176-183, 186-188
bkr.server.watchdog 28
13 54% 18-36, 50-60
bkr.server.widgets 1093
531 51% 43, 50-53, 55, 60, 64-67, 84-85, 93-94, 107, 110-111,
120-130, 152-157, 179-184, 195-205, 273-276, 296-304, 342-347, 362-378,
381-395, 398-440, 456, 459-464, 471, 502-505, 518-522, 554-608, 633-650,
693-708, 712-714, 717, 723-729, 738-772, 775-777, 794-797, 800-803,
854-856, 881-890, 923-931, 991-1023, 1026-1033, 1048-1051, 1071-1075,
1094-1098, 1120-1129, 1166-1177, 1212-1216, 1230-1234, 1257-1259,
1290-1295, 1373-1374, 1378-1390, 1398-1410, 1413-1454, 1468, 1487,
1492-1501, 1513-1514, 1532-1539, 1556-1565, 1577-1578, 1599-1601,
1604-1606, 1627-1628, 1631-1633, 1648-1659, 1666-1667, 1682-1685,
1714-1715, 1734-1735, 1743-1744, 1747-1772, 1794-1800, 1803-1804,
1818-1832, 1842-1844
bkr.server.xmlrpccontroller 46
31 33% 30-44, 48-78
bkr.timeout_xmlrpclib 27
16 41% 5-11, 18-20, 25-30, 36
------------------------------------------------------------------------------------------
TOTAL 30667
8998 71%
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 10 months
Beaker 0.13.1 now available
by Nick Coghlan
We are very pleased to announce that Beaker v0.13.1 in now available
from the yum repos on beaker-project.org, and the main site has been
updated accordingly.
In this release we have introduced the first components of the Enhanced User
Groups design proposal [1]. This includes:
- explicit separation of "individual jobs" and "group jobs".
- support for user defined groups.
- optional LDAP integration for group definitions
Following the migration to Sphinx-based documentation, Beaker 0.13
introduces substantial changes to the upstream Beaker User Guide. These
changes should make it easier for users to get started working with an
existing Beaker installation. The greatest changes can be seen in the
section on writing Beaker tasks [2]. Improvements have also been made to
the job submission and execution documentation [3].
Additional details about the above features and smaller changes for this
release can be found in the release notes here:
http://beaker-project.org/docs/whats-new/release-0.13.html
Thank you to all who contributed to this release.
We're currently working on an incremental maintenance release (0.13.2),
while also starting to think seriously about the scope of Beaker 0.14
(which we're currently aiming to release some time in August).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 10 months
Proposed changes to "System Pools" and "Implicit System Pools" design proposals
by Nick Coghlan
Currently, the "System Pools" and "Implicit System Pools" design
proposals are structured such that the pools are initially made
explicit, and then we add a separate mechanism to implicitly associated
them with users and groups.
http://beaker-project.org/dev/proposals/system-pools.html
http://beaker-project.org/dev/proposals/implicit-system-pools.html
I'd like to propose that we instead consider approach this differently,
and do the *implicit* pools first, using them as a back end
implementation technique for existing Beaker concepts, and then expand
them to additional use cases over time.
With this revised approach, the first increment of the functionality
would look at implicitly associating a system pool with every user. When
a user is the "owner" for a system, then that system is considered part
of their implicitly associated pool
The advantage of this approach is that it allows us to adopt the pool
model for system permissions handling, *without* needing to immediately
introduce new UI concepts for explicit management of the pools. Instead,
users would merely gain the ability to set usage policy for systems they
own.
The second increment of the functionality would then expand the implicit
pool concept to user groups. Associating a system with a group would
then make the system part of that group's pool in addition to being part
of the owner's pool. This increment would need to address some of the
thornier permissions questions that arise when a system is part of two
pools (at the least the owner's pool and the group's pool, but
potentially also pools for other groups as well).
Finally, we will be able to expand the pool functionality to permit the
creation of explicit system groups, separate from the implicit groups,
in order to set special permissions for those systems (or just to ensure
they aren't used accidentally).
This seems like a more natural evolution from the existing capabilities
of Beaker to where we want to get to.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 10 months
Proposed change to "Submission Delegates" design proposal
by Nick Coghlan
Currently, the notion of "Submission Delegates" is part of the Enhanced
User Groups design proposal:
http://beaker-project.org/dev/proposals/enhanced-user-groups.html#submiss...
However, looking at some of the feedback that led to the creation of
that feature, I'm wondering if it actually makes sense for "Submission
Delegates" to be a group level feature. Since the only permission it
grants is the ability to submit jobs, perhaps it would make more sense
for it to be pulled out to a separate design proposal as a *user* level
feature?
Then the job header in the XML could just gain a "user=<username>"
attribute (similar to the just added "group=<groupname>" attribute), and
all we would do on the back end is add an informational
"submitting_user" field to jobs in the database, and allow individual
*users* to nominate other users as submission delegates.
This seems like it would be simpler to implement and manage, and also
better serve the intended use case of allowing an external automated
service that simplifies scheduling and submission of jobs on behalf of
various users.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 10 months
Better handling of pending status updates when scheduling recipes
by Nick Coghlan
Currently, there's a conflict between the mechanism we use to ensure job
status is tracked correctly (marking jobs as dirty until the entire set
can be updated appropriately), and the mechanism we use to try to avoid
deadlocks between recipe sets that are competing for the same systems
(by always favouring recipe sets that already have some systems assigned).
The problem is that by taking the dirty recipes out of the set of those
considered for scheduling, we recreate the opportunity for the recipe
set deadlock (see [1]). Our instance appears to have enough machines
available that we don't hit it too often, but it looks like it may cause
more problems for smaller instances.
Ray has a suggestion for how we might be able to better cope with that:
> I wonder if we can just accept dirty jobs in the schedule_queued_recipes() method.
> As a dirty recipe it will be in the 'Queued' state, and the only state that would change
> to, by making it clean, would be 'Aborted' (or 'Cancelled' if the user cancels the job).
> So if, while it is dirty, the recipe is assigned a system and then later marked 'Aborted',
> the recipe will be cleaned up and the system released as would normally happen with a completed recipe.
After thinking it through a bit, I think we definitely need to do
*something* along these lines. I initially thought getting started on
some aspects of
http://beaker-project.org/dev/proposals/event-driven-scheduler.html
would be a different way to solve the problem, but it turns out that
needs the *exact* same ability to take queued-but-dirty recipes into
account when attempting to schedule recipes on newly available systems.
The one issue I see is that we really *don't* want to be making more
status changes to recipes that are part of a dirty job or we're back
into risking status update conflicts - what we really want is a way to
flag those systems as off limits for the remainder of this pass over the
queue, *without* attempting to schedule the dirty recipe immediately.
Cheers,
Nick.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=952587
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 10 months
Difference in behavior of xml.dom.minidom's toprettyxml() between Python 2.6 and 2.7
by Amit Saha
One of our two test failures on Fedora 18 is this one:
FAIL: bkr.inttest.server.test_jobs:TestJobsController.test_job_xml_can_be_roundtripped
The reason is toprettyxml()'s behavior between Python 2.6 and Python 2.7. From an example at [1], on Python 2.6:
>>> from xml.dom import minidom
>>> d = minidom.parseString('<foo><bar>AAA</bar>BBB<bar>CCC</bar></foo>')
>>> print d.toprettyxml()
<?xml version="1.0" ?>
<foo>
<bar>
AAA
</bar>
BBB
<bar>
CCC
</bar>
</foo>
And on Python 2.7:
>>> from xml.dom import minidom
>>> d = minidom.parseString('<foo><bar>AAA</bar>BBB<bar>CCC</bar></foo>')
>>> d.toprettyxml()
u'<?xml version="1.0" ?>\n<foo>\n\t<bar>AAA</bar>\n\tBBB\n\t<bar>CCC</bar>\n</foo>\n'
>>> print d.toprettyxml()
<?xml version="1.0" ?>
<foo>
<bar>AAA</bar>
BBB
<bar>CCC</bar>
</foo>
I am not sure, but it looks like this is where the "fix" was committed in 2.7/3.x branches[2].
[1] http://ronrothman.com/public/leftbraned/xml-dom-minidom-toprettyxml-and-s...
[2] http://bugs.python.org/issue4147
At this moment, I am not sure what we can do about it, since changing our expected XML file will break stuff on 2.6/RHEL6.
Best,
Amit.
--
Amit Saha <http://echorand.me>
Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 11 months
The march towards Beaker 1.0
by Nick Coghlan
Beaker has been doing 0.x releases for a long time. We've been talking
about the release later this month being 1.0, but before we declare
we've reached that milestone, I'd really like us to be at the point
where someone could reasonably expect to grab the Beaker RPMs from
beaker-project.org, and using just our documentation, set up and start
running their own Beaker instance, including writing new tests using
either beah or autotest.
We're not there yet, and we're not going to get there this month, so the
next release will be 0.13 and we'll continue with regular 0.x releases
until we're happy we have something we consider worthy of the "Beaker
1.0" title. I don't think we're all that far away - we may need a 0.14,
but I'll be surprised if we see 0.15.
For planning purposes, I'd like to keep marking the backlog as "1.0" -
the 0.x releases will then just be snapshots that make useful chunks of
functionality available to users, rather than having to wait until we're
done with everything we want to include for the 1.0 milestone.
This will throw out the currently predicted releases in
http://beaker-project.org/dev/proposals/handling-large-installations.html and
the associated design proposals, but I'm not really worried about that.
In some respects, it works out better, as it means we can add the
"Effective Job Priorities" proposal to our targets for Beaker 1.0, and
that's the *real* objective of several of the changes we're currently
working on.
Once we're happy we know what we would like to include in 1.0, then I
can go back and adjust those design proposals and the tech roadmap
accordingly.
Based on the off-list discussions, here's what I would like to aim for
as our 1.0 milestone:
* The Effective Job Priorities design proposal (and its dependencies)
* Full Fedora compatibility (including the test suite)
* A stable alternate harness API (ideally with the corresponding
autotest patches merged on their side)
* A harness independent reservesys mechanism
* Moving any remaining hardcoded Red Hat specific settings into
configuration files
* Documentation improvements, including
* an architectural guide that at least explains all of Beaker's
external interfaces
* an explanation of the local watchdog hooks
* more general advice on job design (including effectively using
features like the job matrix)
* Self tests that exercise beah, the legacy harness API and the new
harness API
* Having a public page giving at least basic info on how Beaker is used
at Red Hat
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 11 months