Bugzilla Guide¶
Contributor guidelines¶
We have developed several tools/workflows to make the process for dealing with Bugzilla bugs (BZs) easier for contributors. Here, we discuss:
Metadata Markers for test cases:
blockers
,coverage
, andautomates
the miq bz command command
Metadata Markers¶
blockers: this marker is used when a test case is blocked by a bugzilla bug. To use this marker simply add a
@pytest.mark.metadata(blockers=[BZ(<bug_id>])
decorator to your test case. At run time, the propertyBZ(<bug_id>).blocks
will be checked. This property, defined incfme.utils.blockers.py::BZ
will returnTrue
if the BZ is open, not in anON_QA
,VERIFIED
, orCLOSED
state, and if the BZ appliance version matches (or is lower than) the current appliance version you’re testing against. IfBZ().blocks == True
, the test case will be skipped. For example,@pytest.mark.meta(blockers=[BZ(1234567)]) def test_blocked(): """ This test case is blocked by the BZ with id 1234567 """ pass
There are several key word arguments that can be passed to this
BZ
class:unblock
,forced_streams
,ignore_bugs
etc.unblock
takes a boolean function so that the BZ will not block if some condition is met.forced_streams
is to be used when a BZ written against one appliance version is also present in another appliance version (and a clone has not yet been made). It will force the BZ to block in the other appliance version. Note: if a workaround is possible, it is better to putBZ(<bug_id>).blocks
in the test case or broken bit of framework, so that the test case can still run. Any BZs passed toignore_bugs
will not block.coverage: this marker is used to denote manual test cases that are providing coverage for a specific BZ. Test cases that are marked with this will be parsed by the
miq bz
command and can setqe_test_coverage
flags on Bugzilla. If the test case covers more than one BZ, then you can put multiple entries in the list. For example,@pytest.mark.meta(coverage=[1234567, 1234568]) def test_coverage(): """ This test case provides coverage for the BZs 1234567 and 1234568 """ pass
automates: this marker is used to denote automated test cases that are providing coverage for a specific BZ. It is identical to
coverage
but meant for automated test cases. For example,@pytest.mark.meta(automates=[1234567, 1234568]) def test_automated(): """ This test case provides (poor) automated coverage for the BZs 1234567 and 1234568 """ assert True
The markers automates
and coverage
should not be used in the same test case. However,
if a test case is blocked by a BZ and is also providing coverage for that BZ, then you can
use both automates OR coverage
and blockers
. Also note that blockers
must be
passed in as an instance the BZ
class from cfme.utils.blockers
, but for automates
or coverage
you can pass just the id
or an instance of the BZ
class.
miq bz command¶
The miq bz
command is a CLI utility for generating reports on BZs that have coverage.
It looks for test cases that are marked with automates
or coverage
, and gathers
data about the BZ. Most importantly it looks at whether or not qe_test_coverage
is set to
+
, ?
, or -
. When you mark a test case with automates
or coverage
, you are
saying that the test case is providing qe_test_coverage
for a specific BZ. Therefore, qe_test_coverage
should be +
for that BZ.
The help for this command is:
Usage: miq bz [OPTIONS] COMMAND [ARGS]...
Functions for generating reports on BZs included in test suite metadata
Options:
--help Show this message and exit.
Commands:
coverage Set QE test coverage flag based on automates/coverage metadata
list List open/closed BZs that have test coverage
report Generate BZ report on BZs that have coverage given a directory
And the calling sequence is miq bz <command> <directory or test module> <optional-args>
.
For example, say you find a new BZ and write a test case to cover that BZ in
cfme.tests.control.test_bugs
. Here let’s use an actual example from our codebase:
@pytest.mark.meta(blockers=[BZ(1717483)], automates=[1711352])
def test_policy_condition_multiple_ors(
appliance,
virtualcenter_provider,
vm_compliance_policy_profile
):
"""
Tests to make sure that policy conditions with multiple or statements work properly
Bugzilla:
1711352
1717483
Polarion:
assignee: jdupuy
caseimportance: low
casecomponent: Control
initialEstimate: 1/12h
"""
collection = appliance.provider_based_collection(virtualcenter_provider)
all_vms = collection.all()
all_vm_names = [vm.name for vm in all_vms]
# we need to select out cu-24x7
vm_name = virtualcenter_provider.data["cap_and_util"]["capandu_vm"]
# check that it exists on provider
if not virtualcenter_provider.mgmt.does_vm_exist(vm_name):
pytest.skip("No capandu_vm available on virtualcenter_provider of name {}".format(vm_name))
vms = [all_vms.pop(all_vm_names.index(vm_name))]
# do not run the policy simulation against more that 4 VMs
try:
vms.extend(all_vms[0:min(random.randint(1, len(all_vms)), 4)])
except ValueError:
pytest.skip("No other vms exist on provider to run policy simulation against.")
filtered_collection = collection.filter({"names": [vm.name for vm in vms]})
# Now perform the policy simulation
view = navigate_to(filtered_collection, "PolicySimulation")
# Select the correct policy profile
view.fill({"form": {"policy_profile": "{}".format(vm_compliance_policy_profile.description)}})
# Now check each quadicon and ensure that only cu-24x7 is compliant
for entity in view.form.entities.get_all():
state = entity.data["quad"]["bottomRight"]["tooltip"]
if entity.name == vm_name:
assert state == "Policy simulation successful."
else:
assert state == "Policy simulation failed with: false"
This a nice test case because it combines several of the things above. It is blocked
by the BZ 1717483, but it is providing automated test coverage for BZ 1711352. Note that it isn’t
providing coverage for BZ 1717483, so that BZ must be blocking a setup or teardown step of
the test case. It also makes use of the Bugzilla
docblock, discussed below.
You can then run the following command to set qe_test_coverage to +
for BZ 1711352.
miq bz coverage --set cfme/tests/control/test_bugs.py
A dry run of this command (i.e. without --set
) produces the following output:
The following BZs should have qe_test_coverage set to '+':
id: 1155284, qe_test_coverage: ?
id: 1243357, qe_test_coverage: ?
id: 1711352, qe_test_coverage: -
Bugzilla docblock¶
The Bugzilla
docblock is for listing any BZs that are tangentially related to a test case.
Blockers, automates, and coverage BZs should be listed here. Any BZ related to a test case
should also be put here. The BZs listed in this docblock will be shown in the test case
description on Polarion. Therefore, it’s useful to list all the BZs related to the test
case here. For example
@pytest.mark.meta(blockers=[BZ(1234561), automates=[1234567])
def test_automated():
"""
This test case is blocked by BZ 1234561, but provides automated test coverage for
BZ 1234567. The following BZs are related to this test case:
Bugzilla:
1234561
1234567
1234562
"""
assert True
So here in the Bugzilla
docblock, in addition to the two BZs listed in the test case’s
metadata, there is an additional BZ 1234562
which is tangentially related to the test
case. These additional BZs could be in another focus area, but your test case happens to
hit them. They can be useful to someone trying to debug a failing test case.