doc/testing.txt
author Danek Duvall <danek.duvall@oracle.com>
Thu, 08 Sep 2016 17:47:13 -0700
changeset 6910 93182fb9923a
parent 5241 21a30e6ac5fd
permissions -rw-r--r--
24589822 deliver httplib2 for Python 3.x
Summary.
--------

Testing of a userland component that provides tests is performed by hooking
those tests up to the 'test' target and running 'gmake test'. This should
generally pass, as failing tests may indicate things you have to fix, or
upstream tests that aren't applicable or need modifications on Solaris.

'gmake test' is often run when a component is upgraded or otherwise
intentionally changed, but sometimes it would be useful to rerun the tests
after something else has changed (such as the system being upgraded, or a
change in compilers; see the 'System Test' section below) and see if that
has affected the tests.

We do this by having a 'master test file' that contains the expected results,
and having a compare target that runs the tests and compares them with the
results.

Note: because the initial test run and the current test run may have a
different environment (different users, different locales, different machines,
different compilers...) the results need to have all such output dependencies
removed or abstracted.


Setting up a master test file for a Userland component.
-------------------------------------------------------

When setting up a test-and-compare run for a new component, you will need to
have master test file(s) that are identical for both the x86 and SPARC
platforms. It is suggested that they should initially be created by doing
something like the following in your x86 Userland workspace, then copied to
the same locations in your SPARC workspace and retested there.

When you run "gmake test", a check is made to see if there is a master
file of test results. If there is, then a test-then-compare run is performed.
If there isn't, then just a "normal" run of the test suite is performed.

The name of the master test file (or files to be exact), will depend upon
whether you have 32-bit, 64-bit, 32-and-64-bit and whether this is for a
Python or Perl component.

The default master file name is defined in
make-rules/shared-macros.mk and is:

COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(BITS).master

where COMPONENT_TEST_RESULTS_DIR is:

COMPONENT_TEST_RESULTS_DIR =    $(COMPONENT_DIR)/test

so that means it will default to looking for the following test file
master names:

32-bit: components/<component-name>/test/results-32.master

64-bit: components/<component-name>/test/results-64.master

both:   components/<component-name>/test/results-32.master
        components/<component-name>/test/results-64.master

For Python, COMPONENT_TEST_MASTER is overridden in
make-rules/setup.py.mk to be:

COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PYTHON_VERSION)-$(BITS).master

so that means it's looking for one or more of:

2.6:   components/python/<component-name>/test/results-2.6-32.master
2.6:   components/python/<component-name>/test/results-2.6-64.master
2.7:   components/python/<component-name>/test/results-2.7-32.master
2.7:   components/python/<component-name>/test/results-2.7-64.master
3.4:   components/python/<component-name>/test/results-3.4-64.master

depending upon which versions of Python this component supports.

Perl is similar, with COMPONENT_TEST_MASTER being overridden in:
make-rules/makemaker.mk to be:

COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PERL_VERSION)-$(BITS).master

so that means it's looking for one or more of:

5.12:    components/perl_modules/<component>/test/results-5.12-32.master
5.12-mt: components/perl_modules/<component>/test/results-5.12-mt-32.master
5.16:    components/perl_modules/<component>/test/results-5.16-64.master

depending upon which versions of Perl this component supports.

Note that if the test results are the same for both 32-bit and 64-bit or
for all versions of Python or Perl, you can override the
COMPONENT_TEST_MASTER definition in your component Makefile and just supply
a single master files file. For example, in components/python/pep8 Makefile
we have:

COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-all.master

In order to do a test-then-compare run rather than just run the component
test suite, initially just create an empty master test file (or files).

For example, for elinks, which just has 64-bit tests, do:

   $ cd components/elinks
   $ mkdir test
   $ touch test/results-64.master
   $ gmake test

At this point, you have a set of test results in
components/elinks/build/test/$(MACH64)/test-64-results.

Even better, there are a set of "global" regular expressions that are
run on those test results to try to normalize them. The output from that
is placed in components/elinks/build/test/$(MACH64)/results-64.snapshot

You can now use the contents of that file as a first cut at the master results.

   $ cp build/test/$(MACH64)/results-64.snapshot test/results-64.master

Now run the tests again. Note that you have to get back to a clean start
just in case the test process tries to compile any code.

   $ gmake clean
   $ gmake test

At this point, it will again compare the test results against the master(s),
and if there are still differences, they will be placed in
components/elinks/build/test/$(MACH64)/test-64-diffs

Typically these differences will be for things like usernames, temporary
filenames, timings etc. If you have some differences, then you are going
to have to write special regexp expressions in your component Makefile to
"normalize" them, and adjust the master test results file so that it
matches what the normalized version looks like.

Transform examples.
-------------------

1/ Handling temporary filenames.

See the transform in the asciidoc Makefile:

COMPONENT_TEST_TRANSFORMS += \
        '-e "s|/tmp......|/tmpxxxxxx|g" '

for how to "normalize" any temporary filenames that appear in the test results.

2/ Build related output.

Items (such as paths to compilers) should be removed from master test files.
For example, in testing the xz component, we get the following different
output for compiling a .c file for 32-bit and 64-bit on x86 and SPARC:

x86 32-bit:

/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I..  -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib  -D_REENTRANT  -m32 -xO4 -xchip=pentium -xregs=no%frameptr    -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c

x86 64-bit:

/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I..  -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib  -D_REENTRANT  -m64 -xO4 -xchip=generic -Ui386 -U__i386 -xregs=no%frameptr    -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c

SPARC 32-bit:

/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I..  -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib  -D_REENTRANT  -m32 -xO4 -xtarget=ultra2 -xarch=sparcvis -xchip=ultra2 -Qoption cg -xregs=no%appl -W2,-xwrap_int  -xmemalign=8s -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c

SPARC 64-bit:

/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I..  -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib  -D_REENTRANT  -m64 -xO4  -xtarget=ultra2 -xarch=sparcvis -xchip=ultra2 -Qoption cg -xregs=no%appl -W2,-xwrap_int  -xmemalign=16s -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c

These can all be normalized (so that a single master test file can be used),
by adding the following line to COMPONENT_TEST_TRANSFORMS in the xz component
Makefile:

	'-e "s|\(^$(CC)\).*|\\$$(CC)|" '

The equivalent compile line in the master test file is now set to just:

$(CC)

Note that we are using $(CC) here which will pick up the current C compiler.
Should the C compiler or build options change in the future, then there will
be no need to adjust the master test results file to match. It's all done
automatically.

It is recognized that an alternate approach to solving this problem is to
create a transform to just remove lines like this from the master test files,
but there is currently no automatic way to pick up the default compiler
(see CR #20367119 for more details).

----

There will be other examples as more components are adjusted to test
against master results files.

If a lot of people start generating the very same ones, then we can
consider adding them to the "global" set of regexps in
make-rules/shared-macros.mk which currently looks like:

COMPONENT_TEST_TRANSFORMS = \
        '-e "s|$(@D)|\\$$(@D)|g" ' \
        '-e "s|$(PERL)|\\$$(PERL)|g" ' \
        '-e "s|$(SOURCE_DIR)|\\$$(SOURCE_DIR)|g" '

Note that in some cases, gsed is not powerful enough.  E.g., if you need to
do more than massage or remove certain lines, but only compare certain sections
of the file, then you may need to set COMPONENT_TEST_TRANSFORMER to something
other than $(GSED) such as $(NAWK).  In this case, you will want to set
COMPONENT_TEST_TRANSFORMS using "=" instead of "+=", as the default sed
transforms will likely need to be tossed aside in favor of whatever arguments
your favorite transformer might need.

When your master test file(s) are in good shape, then you should "hg add"
them to your workspace.


System Test
-----------
All of the above discusses how 'gmake test' works.  This is all good, but
has the limitation that objects under test are those that were just built
and installed into the component's proto area.  We also want to be able
to test whatever is installed on the actual system.  This has the benefits
of not requiring anything to be built, and allowing non-Userland people
to test our bits.  (Think of people in ON changing libc or the linker and
wanting to make sure they don't break anything.)

To this end, we have added 'gmake system-test'.  It works just like 'gmake
test', reusing some of the same variables:
* COMPONENT_TEST_BUILD_DIR
* COMPONENT_TEST_COMPARE
* COMPONENT_TEST_CREATE_TRANSFORMS
* COMPONENT_TEST_OUTPUT
* COMPONENT_TEST_PERFORM_TRANSFORM
but with its own "_SYSTEM" instance of other variables:
* COMPONENT_POST_SYSTEM_TEST_ACTION
* COMPONENT_PRE_SYSTEM_TEST_ACTION
* COMPONENT_SYSTEM_TEST_ARGS
* COMPONENT_SYSTEM_TEST_CLEANUP
* COMPONENT_SYSTEM_TEST_CMD
* COMPONENT_SYSTEM_TEST_DIR
* COMPONENT_SYSTEM_TEST_ENV
* COMPONENT_SYSTEM_TEST_ENV_CMD
* COMPONENT_SYSTEM_TEST_RESULTS_DIR
* COMPONENT_SYSTEM_TEST_TARGETS

In the ideal case, only Makefile variables would need to be modified to
take a component where 'gmake test' works and extend it so that 'gmake
system-test' works; see components/python34/Makefile for such an example.

The next level up from that is the case where some simple patching is needed.
In this case, simple means:
* tweaking LD_LIBRARY_PATH
* changing the path to a binary to run
* modifying a Makefile so tests are compiled as part of 'gmake build'
* modifying a Makefile so tests are installed as part of 'gmake install'
* modifying a pkg(5) manifest so tests are published as part of 'gmake publish'
  (preferably under the "optional.test" facet)

If more extensive patching is needed, a judgement call needs to be made:
Are these changes that upstream would welcome?  If not, then we would need
to keep such changes around indefinitely, which would constitute a fork,
which we try very hard to avoid.  When 'gmake system-test' was initially
implemented, changes to both fetchmail and python/decorator were begun but
ultimately abandoned for this reason; see 20808505 for details.