# HG changeset patch # User Rich Burridge # Date 1421425229 28800 # Node ID ebe894a8833efc554fcfdf8683019a222513ae02 # Parent f8449ec64ef3dfef4a24520bf42713b75fbf1681 20359322 Need documentation for adding master test results to Userland components diff -r f8449ec64ef3 -r ebe894a8833e doc/testing.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/testing.txt Fri Jan 16 08:20:29 2015 -0800 @@ -0,0 +1,150 @@ +Summary. +-------- + +Testing of a userland component that provides tests is performed by hooking +those tests up to the 'test' target and running 'gmake test'. This should +generally pass, as failing tests may indicate things you have to fix, or +upstream tests that aren't applicable or need modifications on Solaris. + +'gmake test' is often run when a component is upgraded or otherwise +intentionally changed, but sometimes it would be useful to rerun the tests +after something else has changed (such as the system being upgraded, or a +change in compilers) and see if that has affected the tests. + +We do this by having a 'master test file' that contains the expected results, +and having a compare target that runs the tests and compares them with the +results. + +Note: because the initial test run and the current test run may have a +different environment (different users, different locales, different machines, +different compilers...) the results need to have all such output dependencies +removed or abstracted. + + +Setting up a master test file for a Userland component. +------------------------------------------------------- + +When setting up a test-and-compare run for a new component, +you should be able to do something like the following. + +When you run "gmake test", a check is made to see if there is a master +file of test results. If there is, then a test-then-compare run is performed. +If there isn't, then just a "normal" run of the test suite is performed. + +The name of the master test file (or files to be exact), will depend upon +whether you have 32-bit, 64-bit, 32-and-64-bit and whether this is for a +Python or Perl component. + +The default master file name is defined in +make-rules/shared-macros.mk and is: + +COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(BITS).master + +where COMPONENT_TEST_RESULTS_DIR is: + +COMPONENT_TEST_RESULTS_DIR = $(COMPONENT_DIR)/test + +so that means it will default to looking for the following test file +master names: + +32-bit: components//test/results-32.master + +64-bit: components//test/results-64.master + +both: components//test/results-32.master + components//test/results-64.master + +For Python, COMPONENT_TEST_MASTER is overridden in +make-rules/setup.py.mk to be: + +COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PYTHON_VERSION)-$(BITS).master + +so that means it's looking for one or more of: + +2.6: components/python//test/results-2.6-32.master +2.6: components/python//test/results-2.6-64.master +2.7: components/python//test/results-2.7-32.master +2.7: components/python//test/results-3.4-64.master +3.4: components/python//test/results-3.4-64.master + +depending upon which versions of Python this component supports. + +Perl is similar, with COMPONENT_TEST_MASTER being overridden in: +make-rules/makemaker.mk to be: + +COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PERL_VERSION)-$(BITS).master + +so that means it's looking for one or more of: + +5.12: components/perl_modules//test/results-5.12-32.master +5.12-mt: components/perl_modules//test/results-5.12-mt-32.master +5.16: components/perl_modules//test/results-5.16-64.master + +depending upon which versions of Perl this component supports. + +Note that if the test results are the same for both 32-bit and 64-bit or +for all versions of Python or Perl, you can override the +COMPONENT_TEST_MASTER definition in your component Makefile and just supply +a single master files file. For example, in components/python/pep8 Makefile +we have: + +COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-all.master + +In order to do a test-then-compare run rather than just run the component +test suite, initially just create an empty master test file (or files). + +For example, for elinks, which just has 64-bit tests, do: + + $ cd components/elinks + $ mkdir test + $ touch test/results-64.master + $ gmake test + +At this point, you have a set of test results in +components/elinks/test/test-64-results. + +Even better, there are a set of "global" regular expressions that are +run on those test results to try to normalize them. The output from that +is placed in components/elinks/test/results-64.snapshot + +You can now use the contents of that file as a first cut at the master results. + + $ cp test/results-64.snapshot test/results-64.master + +Now run the tests again. Note that you have to get back to a clean start +just in case the test process tries to compile any code. + + $ gmake clean + $ gmake test + +At this point, it will again compare the test results against the master(s), +and if there are still differences, they will be placed in +components/elinks/test/test-64-diffs + +Typically these differences will be for things like usernames, temporary +filenames, timings etc. If you have some differences, then you are going +to have to write special regexp expressions in your component Makefile to +"normalize" them, and adjust the master test results file so that it +matches what the normalized version looks like. + +For example, see the transform in the asciidoc Makefile: + +COMPONENT_TEST_TRANSFORMS += \ + '-e "s|/tmp......|/tmpxxxxxx|g" ' + +to "normalize" any temporary filenames that appear in the test results. + +There will be other examples as more components are adjusted to test +against master results files. + +If a lot of people start generating the very same ones, then we can +consider adding them to the "global" set of regexps in +make-rules/shared-macros.mk which currently looks like: + +COMPONENT_TEST_TRANSFORMS = \ + '-e "s|$(@D)|\\$$(@D)|g" ' \ + '-e "s|$(PERL)|\\$$(PERL)|g" ' \ + '-e "s|$(SOURCE_DIR)|\\$$(SOURCE_DIR)|g" ' + +When your master test file(s) are in good shape, then you should "hg add" +them to your workspace.