qm.test.classes.tet

32 """A 'TETStream' formats results as a TET journal. 33 34 Provides special handling for 'DejaGNUTest' results. 35 36 TET: http://tetworks.opengroup.org/ 37 TET journal format: see appendix C and D of 38 http://tetworks.opengroup.org/documents/3.7/uguide.pdf 39 40 For the meaning of TET result codes, we use as guidelines the LSB 41 test faq, question Q1.11: 42 * PASS - a test result belonging to this group is considered to 43 be a pass for compliance testing purposes: 44 o Pass - the test has been executed correctly and to 45 completion without any kind of problem 46 o Warning - the functionality is acceptable, but you 47 should be aware that later revisions of the relevant 48 standards or specification may change the requirements 49 in this area. 50 o FIP - additional information is provided which needs to 51 be checked manually. 52 o Unsupported - an optional feature is not available or 53 not supported in the implementation under test. 54 o Not in Use - some tests may not be required in certain 55 test modes or when an interface can be implemented by a 56 macro or function and there are two versions of the test 57 only one is used. 58 o Untested - no test written to check a particular feature 59 or an optional facility needed to perform a test is not 60 available on the system. 61 [There are also "notimp" and "unapproved" cases mentioned in 62 the LSB-FHS README, but they are otherwise undocumented, and 63 don't correspond to any DejaGNU or QMTest outcomes anyway.] 64 * FAIL - a test result belonging to this group is considered to 65 be a fail for compliance testing purposes (unless the failure 66 has been waived by an agreed Problem Report in the 67 Certification Problem Reporting database): 68 o Fail - the interface did not behave as expected. 69 o Uninitiated - the particular test in question did not 70 start to execute. 71 o Unresolved - the test started but did not reach the 72 point where the test was able to report success or 73 failure. 74 o Unreported - a major error occurred during the testset 75 execution. (The TET manual calls this NORESULT.) 76 (From http://www.linuxbase.org/test/lsb-runtime-test-faq.html ) 77 78 DejaGNU test results are described as: 79 * PASS - A test has succeeded. 80 * FAIL - A test has produced the bug it was intended to 81 capture. 82 * WARNING - Declares detection of a minor error in the test case 83 itself. Use WARNING rather than ERROR for cases (such as 84 communication failure to be followed by a retry) where the 85 test case can recover from the error. Note that sufficient 86 warnings will cause a test to go from PASS/FAIL to 87 UNRESOLVED. 88 * ERROR - Declares a severe error in the testing framework 89 itself. An ERROR also causes a test to go from PASS/FAIL to 90 UNRESOLVED. 91 * UNRESOLVED - A test produced indeterminate results. Usually, 92 this means the test executed in an unexpected fashion; this 93 outcome requires that a human being go over results, to 94 determine if the test should have passed or failed. This 95 message is also used for any test that requires human 96 intervention because it is beyond the abilities of the testing 97 framework. Any unresolved test should be resolved to PASS or 98 FAIL before a test run can be considered finished. 99 100 Examples: 101 - a test's execution is interrupted 102 - a test does not produce a clear result (because of 103 WARNING or ERROR messages) 104 - a test depends on a previous test case which failed 105 * UNTESTED - a test case that isn't run for some technical 106 reason. (E.g., a dummy test created as a placeholder for a 107 test that is not yet written.) 108 * UNSUPPORTED - Declares that a test case depends on some 109 facility that does not exist in the testing environment; the 110 test is simply meaningless. 111 (From a combination of DejaGNU manual sections "Core Internal 112 Procedures", "C Unit Testing API", and "A POSIX conforming test 113 framework".) 114 115 """ 116 117 # TET result codes: 118 PASS = (0, "PASS") 119 WARNING = (101, "WARNING") 120 FIP = (102, "FIP") 121 UNSUPPORTED = (4, "UNSUPPORTED") 122 NOTINUSE = (3, "NOTINUSE") 123 UNTESTED = (5, "UNTESTED") 124 125 FAIL = (1, "FAIL") 126 UNINITIATED = (6, "UNINITIATED") 127 UNRESOLVED = (2, "UNRESOLVED") 128 UNREPORTED = (7, "UNREPORTED") 129 130

131 - def __init__(self, arguments = None, **args):

132 133 super(TETStream, self).__init__(arguments, **args) 134 135 self._start_time = "<unknown_start_time>" 136 self._finish_time = "<unknown_finish_time>" 137 self._aborted = False 138 self._username = "<unknown_user>" 139 self._userid = "<unknown_user>" 140 self._version = "<unknown_version>" 141 self._uname = "<unknown_uname>" 142 self._cmdline = "<unknown_command_line>" 143 self._settings = {} 144 145 self._tcc_number = 0 146 self._printed_initial_stuff = False

147 148

149 - def _WriteLine(self, code, data, comment):

150 """Write a line in TET journal format.""" 151 152 self.file.write("%i|%s|%s\n" % (code, data, comment))

153 154

155 - def _IsDejaGNUResult(self, result):

156 """Returns 'True' if 'result' has DejaGNU subtests.""" 157 158 for key in result.keys(): 159 if key.startswith(DejaGNUTest.RESULT_PREFIX): 160 return True 161 return False

162 163

164 - def _TETFormatTime(self, time_string):

165 """Converts an ISO-format date-time to a TET-format date-time. 166 167 returns -- A 2-tuple whose first element is the time as a string, 168 and whose second is the date as a string.""" 169 170 t = time.gmtime(qm.common.parse_time_iso(time_string)) 171 172 return (time.strftime("%H:%M:%S", t), 173 time.strftime("%Y%m%d", t))

174 175

176 - def _ExtractTime(self, result, key):

177 """Extracts the start time from a result.""" 178 179 if result.has_key(key): 180 return self._TETFormatTime(result[key])[0] 181 else: 182 return "00:00:00"

183 184

185 - def WriteAnnotation(self, key, value):

186 187 if key == "qmtest.run.start_time": 188 self._start_time, self._start_date \ 189 = self._TETFormatTime(value) 190 elif key == "qmtest.run.end_time": 191 self._finish_time, self._finish_data \ 192 = self._TETFormatTime(value) 193 elif key == "qmtest.run.aborted" and value == "true": 194 self._aborted = True 195 elif key == "qmtest.run.username": 196 self._username = value 197 elif key == "qmtest.run.userid": 198 self._userid = value 199 elif key == "qmtest.run.version": 200 self._version = "qmtest-" + value 201 elif key == "qmtest.run.uname": 202 self._uname = value 203 elif key == "qmtest.run.command_line": 204 self._cmdline = value 205 else: 206 self._settings[key] = value

207 208

209 - def _WriteInitialStuff(self):

210 """Print TET header information, but only on first call. 211 212 Second and later calls are no-ops.""" 213 214 if self._printed_initial_stuff: 215 return 216 217 # Test case controller start 218 # 0 | version time date | who 219 # who is 220 # "User: <username> (<numeric-id>) TCC Start, Command line: <cmdline>" 221 data = "%s %s %s" % (self._version, 222 self._start_time, 223 self._start_date) 224 who = "User: %s (%s) TCC Start, Command line: %s" \ 225 % (self._username, self._userid, self._cmdline) 226 227 self._WriteLine(0, data, who) 228 # Local system information 229 # 5 | sysname nodename release version machine | text 230 self._WriteLine(5, self._uname, "") 231 # Local system configuration start 232 # 20 | pathname mode | text 233 self._WriteLine(20, "qmtest -1", "Config Start") 234 for item in self._settings.iteritems(): 235 # Configuration variable setting 236 # 30 || variable=value 237 self._WriteLine(30, "", "%s=%s" % item) 238 # Configuration end 239 # 40 || text 240 self._WriteLine(40, "", "Config End") 241 242 self._printed_initial_stuff = True

243 244

245 - def WriteResult(self, result):

246 247 self._WriteInitialStuff() 248 if result.GetKind() == Result.TEST: 249 self._tcc_number += 1 250 if self._IsDejaGNUResult(result): 251 self._WriteDejaGNUResult(result) 252 else: 253 self._WriteTestResult(result) 254 else: 255 # We have a resource result. 256 self._WriteResourceResult(result)

257 258

259 - def _WriteTCStart(self, result):

260 """Write a TET test case start line.""" 261 262 # Test case start 263 # 10 | activity_number testcase_path time | invocable_components 264 started = self._ExtractTime(result, Result.START_TIME) 265 data = "%i %s %s" % (self._tcc_number, 266 "/" + result.GetId(), 267 started) 268 self._WriteLine(10, data, "TC Start")

269 270

271 - def _WriteResultAnnotations(self, result, purpose, 272 num_restrict=None, seq_start=1):

273 """Writes out annotations for a 'result' in TET format. 274 275 Annotations are represented as (sequences of) "test case 276 information" lines. 277 278 'result' -- The 'Result' whose annotations should be written. 279 280 'num_restrict' -- Only write out annotations that end with this 281 number. If the number is '1', also writes out all results that 282 don't end in any number, with "INFO: " prefixed. If 'None', 283 writes out all annotations. 284 285 'seq_start' -- The TET test case information sequence number to 286 start with.""" 287 288 seqnum = seq_start 289 keys = result.keys() 290 keys.sort() 291 for key in keys: 292 value = result[key] 293 prefix = "" 294 if num_restrict is not None: 295 if num_restrict == 1 and key[-1] not in "0123456789": 296 prefix = "INFO: " 297 elif not key.endswith("_%i" % num_restrict): 298 continue 299 300 text = qm.common.html_to_text(value) 301 for line in text.split("\n"): 302 # Test case information 303 # 520 | activity_num tp_num context block sequence | text 304 # 305 # We always set 'tp_num' to zero, because annotations 306 # for us are associated with test cases, not test 307 # purposes. 308 # 'context' is to distinguish text coming from different 309 # subprocesses making up the test purpose; it's 310 # generally the pid. For us, it's always zero. 311 # 'block' is entirely undocumented, and the examples 312 # have it always set to one, so we simply set it to 313 # one as well. 314 # 'sequence' appears to be incremented for each line 315 # within a single test purpose and context. 316 self._WriteLine(520, 317 "%i %i 0 1 %i" % (self._tcc_number, 318 purpose, 319 seqnum), 320 "%s%s: %s" % (prefix, key, line)) 321 seqnum += 1

322 323

324 - def _WriteDejaGNUResult(self, result):

325 """Write out a result that has DejaGNU subtest information.""" 326 327 self._WriteTCStart(result) 328 329 # Get the DejaGNU annotations in sorted order. 330 keys = filter(lambda k: k.startswith(DejaGNUTest.RESULT_PREFIX), 331 result.keys()) 332 keys.sort(lambda k1, k2: cmp(int(k1[len(DejaGNUTest.RESULT_PREFIX):]), 333 int(k2[len(DejaGNUTest.RESULT_PREFIX):]))) 334 335 start_time = self._ExtractTime(result, Result.START_TIME) 336 end_time = self._ExtractTime(result, Result.END_TIME) 337 338 purpose = 1 339 for k in keys: 340 r = result[k] 341 outcome = r[:r.find(":")] 342 # Test purpose start 343 # 200 | activity_number test_purpose_number time | text 344 self._WriteLine(200, 345 "%i %i %s" 346 % (self._tcc_number, purpose, start_time), 347 "TP Start") 348 349 outcome_num, outcome_name \ 350 = { DejaGNUTest.PASS: self.PASS, 351 DejaGNUTest.XPASS: self.PASS, 352 DejaGNUTest.FAIL: self.FAIL, 353 DejaGNUTest.XFAIL: self.FAIL, 354 DejaGNUTest.UNTESTED: self.UNTESTED, 355 DejaGNUTest.UNRESOLVED: self.UNRESOLVED, 356 DejaGNUTest.ERROR: self.UNRESOLVED, 357 DejaGNUTest.WARNING: self.WARNING, 358 # TET's UNSUPPORTED is like a FAIL for tests 359 # that check for optional features; UNTESTED is 360 # the correct correspondent for DejaGNU's 361 # UNSUPPORTED. 362 DejaGNUTest.UNSUPPORTED: self.UNTESTED, 363 }[outcome] 364 # As a special case, check for magic annotation. 365 if result.has_key("test_not_relevant_to_testing_mode"): 366 outcome_num, outcome_name = self.NOTINUSE 367 368 # Write per-purpose annotations: 369 self._WriteResultAnnotations(result, purpose, 370 num_restrict=purpose) 371 372 # Test purpose result 373 # 220 | activity_number tp_number result time | result-name 374 data = "%i %i %i %s" % (self._tcc_number, 375 purpose, 376 outcome_num, 377 end_time) 378 self._WriteLine(220, data, outcome_name) 379 380 purpose += 1 381 382 # Test case end 383 # 80 | activity_number completion_status time | text 384 # "completion status" appears undocumented; it is zero in all of 385 # the documented examples. 386 self._WriteLine(80, 387 "%i 0 %s" % (self._tcc_number, end_time), 388 "TC End")

389 390

391 - def _WriteTestResult(self, result):

392 """Write out a result that does not have DejaGNU annotations.""" 393 394 self._WriteTCStart(result) 395 # Test purpose start 396 # 200 | activity_number test_purpose_number time | text 397 start_time = self._ExtractTime(result, Result.START_TIME) 398 data = "%i 1 %s" % (self._tcc_number, start_time) 399 self._WriteLine(200, data, "TP Start") 400 401 outcome_num, outcome_name = { Result.FAIL: self.FAIL, 402 Result.PASS: self.PASS, 403 Result.UNTESTED: self.UNINITIATED, 404 Result.ERROR: self.UNREPORTED, 405 }[result.GetOutcome()] 406 if result.GetOutcome() == Result.ERROR: 407 # Test case information 408 # 520 | activity_num tp_num context block sequence | text 409 # (see _WriteResultAnnotations for details) 410 self._WriteLine(520, 411 "%i 1 0 1 1" % self._tcc_number, 412 "QMTest ERROR in test " + result.GetId()) 413 self._WriteResultAnnotations(result, 1, seq_start=2) 414 else: 415 self._WriteResultAnnotations(result, 1) 416 417 # Test purpose result 418 # 220 | activity_number tp_number result time | result-name 419 end_time = self._ExtractTime(result, Result.END_TIME) 420 data = "%i 1 %i %s" % (self._tcc_number, outcome_num, end_time) 421 self._WriteLine(220, data, outcome_name) 422 423 # Test case end 424 # 80 | activity_number completion_status time | text 425 # "completion status" appears undocumented; it is zero in all of 426 # the documented examples. 427 self._WriteLine(80, 428 "%i 0 %s" % (self._tcc_number, end_time), 429 "TC End")

430 431

432 - def _WriteResourceResult(self, result):

433 """Write out information on a resource result. 434 435 TET has no concept of resources, so we ignore successful 436 resources, and print out "test case controller messages" for 437 ERRORs and FAILUREs.""" 438 439 if result.GetOutcome() in (Result.FAIL, Result.ERROR): 440 if result.GetKind() == Result.RESOURCE_SETUP: 441 verbing = "setting up" 442 elif result.GetKind() == Result.RESOURCE_CLEANUP: 443 verbing = "cleaning up" 444 else: 445 assert False, "Unexpected result kind" 446 id = result.GetId() 447 outcome = result.GetOutcome() 448 # Test case controller message 449 # 50 || text describing problem 450 self._WriteLine(50, "", "Problem with %s resource %s: %s" 451 % (verbing, id, outcome)) 452 453 for key, value in result.items(): 454 for line in value.split("\n"): 455 self._WriteLine(50, "", "%s: %s" % (key, line))

456 457

458 - def Summarize(self):

459 460 self._WriteInitialStuff() 461 462 if self._aborted: 463 # User abort 464 # 90 | time | text 465 self._WriteLine(90, self._finish_time, "Aborted.") 466 467 # Test case controller end 468 # 900 | time | text 469 self._WriteLine(900, self._finish_time, "Done.")

Source Code for Module qm.test.classes.tet_stream