doc/testing.txt
changeset 3640 ebe894a8833e
child 3716 a2629a2cf270
equal deleted inserted replaced
3639:f8449ec64ef3 3640:ebe894a8833e
       
     1 Summary.
       
     2 --------
       
     3 
       
     4 Testing of a userland component that provides tests is performed by hooking
       
     5 those tests up to the 'test' target and running 'gmake test'. This should
       
     6 generally pass, as failing tests may indicate things you have to fix, or
       
     7 upstream tests that aren't applicable or need modifications on Solaris.
       
     8 
       
     9 'gmake test' is often run when a component is upgraded or otherwise
       
    10 intentionally changed, but sometimes it would be useful to rerun the tests
       
    11 after something else has changed (such as the system being upgraded, or a
       
    12 change in compilers) and see if that has affected the tests.
       
    13 
       
    14 We do this by having a 'master test file' that contains the expected results,
       
    15 and having a compare target that runs the tests and compares them with the
       
    16 results.
       
    17 
       
    18 Note: because the initial test run and the current test run may have a
       
    19 different environment (different users, different locales, different machines,
       
    20 different compilers...) the results need to have all such output dependencies
       
    21 removed or abstracted.
       
    22 
       
    23 
       
    24 Setting up a master test file for a Userland component.
       
    25 -------------------------------------------------------
       
    26 
       
    27 When setting up a test-and-compare run for a new component,
       
    28 you should be able to do something like the following.
       
    29 
       
    30 When you run "gmake test", a check is made to see if there is a master
       
    31 file of test results. If there is, then a test-then-compare run is performed.
       
    32 If there isn't, then just a "normal" run of the test suite is performed.
       
    33 
       
    34 The name of the master test file (or files to be exact), will depend upon
       
    35 whether you have 32-bit, 64-bit, 32-and-64-bit and whether this is for a
       
    36 Python or Perl component.
       
    37 
       
    38 The default master file name is defined in
       
    39 make-rules/shared-macros.mk and is:
       
    40 
       
    41 COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(BITS).master
       
    42 
       
    43 where COMPONENT_TEST_RESULTS_DIR is:
       
    44 
       
    45 COMPONENT_TEST_RESULTS_DIR =    $(COMPONENT_DIR)/test
       
    46 
       
    47 so that means it will default to looking for the following test file
       
    48 master names:
       
    49 
       
    50 32-bit: components/<component-name>/test/results-32.master
       
    51 
       
    52 64-bit: components/<component-name>/test/results-64.master
       
    53 
       
    54 both:   components/<component-name>/test/results-32.master
       
    55         components/<component-name>/test/results-64.master
       
    56 
       
    57 For Python, COMPONENT_TEST_MASTER is overridden in
       
    58 make-rules/setup.py.mk to be:
       
    59 
       
    60 COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PYTHON_VERSION)-$(BITS).master
       
    61 
       
    62 so that means it's looking for one or more of:
       
    63 
       
    64 2.6:   components/python/<component-name>/test/results-2.6-32.master
       
    65 2.6:   components/python/<component-name>/test/results-2.6-64.master
       
    66 2.7:   components/python/<component-name>/test/results-2.7-32.master
       
    67 2.7:   components/python/<component-name>/test/results-3.4-64.master
       
    68 3.4:   components/python/<component-name>/test/results-3.4-64.master
       
    69 
       
    70 depending upon which versions of Python this component supports.
       
    71 
       
    72 Perl is similar, with COMPONENT_TEST_MASTER being overridden in:
       
    73 make-rules/makemaker.mk to be:
       
    74 
       
    75 COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PERL_VERSION)-$(BITS).master
       
    76 
       
    77 so that means it's looking for one or more of:
       
    78 
       
    79 5.12:    components/perl_modules/<component>/test/results-5.12-32.master
       
    80 5.12-mt: components/perl_modules/<component>/test/results-5.12-mt-32.master
       
    81 5.16:    components/perl_modules/<component>/test/results-5.16-64.master
       
    82 
       
    83 depending upon which versions of Perl this component supports.
       
    84 
       
    85 Note that if the test results are the same for both 32-bit and 64-bit or
       
    86 for all versions of Python or Perl, you can override the
       
    87 COMPONENT_TEST_MASTER definition in your component Makefile and just supply
       
    88 a single master files file. For example, in components/python/pep8 Makefile
       
    89 we have:
       
    90 
       
    91 COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-all.master
       
    92 
       
    93 In order to do a test-then-compare run rather than just run the component
       
    94 test suite, initially just create an empty master test file (or files).
       
    95 
       
    96 For example, for elinks, which just has 64-bit tests, do:
       
    97 
       
    98    $ cd components/elinks
       
    99    $ mkdir test
       
   100    $ touch test/results-64.master
       
   101    $ gmake test
       
   102 
       
   103 At this point, you have a set of test results in
       
   104 components/elinks/test/test-64-results.
       
   105 
       
   106 Even better, there are a set of "global" regular expressions that are
       
   107 run on those test results to try to normalize them. The output from that
       
   108 is placed in components/elinks/test/results-64.snapshot
       
   109 
       
   110 You can now use the contents of that file as a first cut at the master results.
       
   111 
       
   112    $ cp test/results-64.snapshot test/results-64.master
       
   113 
       
   114 Now run the tests again. Note that you have to get back to a clean start
       
   115 just in case the test process tries to compile any code.
       
   116 
       
   117    $ gmake clean
       
   118    $ gmake test
       
   119 
       
   120 At this point, it will again compare the test results against the master(s),
       
   121 and if there are still differences, they will be placed in
       
   122 components/elinks/test/test-64-diffs
       
   123 
       
   124 Typically these differences will be for things like usernames, temporary
       
   125 filenames, timings etc. If you have some differences, then you are going
       
   126 to have to write special regexp expressions in your component Makefile to
       
   127 "normalize" them, and adjust the master test results file so that it
       
   128 matches what the normalized version looks like.
       
   129 
       
   130 For example, see the transform in the asciidoc Makefile:
       
   131 
       
   132 COMPONENT_TEST_TRANSFORMS += \
       
   133         '-e "s|/tmp......|/tmpxxxxxx|g" '
       
   134 
       
   135 to "normalize" any temporary filenames that appear in the test results.
       
   136 
       
   137 There will be other examples as more components are adjusted to test
       
   138 against master results files.
       
   139 
       
   140 If a lot of people start generating the very same ones, then we can
       
   141 consider adding them to the "global" set of regexps in
       
   142 make-rules/shared-macros.mk which currently looks like:
       
   143 
       
   144 COMPONENT_TEST_TRANSFORMS = \
       
   145         '-e "s|$(@D)|\\$$(@D)|g" ' \
       
   146         '-e "s|$(PERL)|\\$$(PERL)|g" ' \
       
   147         '-e "s|$(SOURCE_DIR)|\\$$(SOURCE_DIR)|g" '
       
   148 
       
   149 When your master test file(s) are in good shape, then you should "hg add"
       
   150 them to your workspace.