doc/testing.txt
author Mike Sullivan <Mike.Sullivan@Oracle.COM>
Thu, 16 Jun 2016 23:00:26 -0700
changeset 6236 9ecffda25374
parent 5241 21a30e6ac5fd
permissions -rw-r--r--
Added tag s12-102 for changeset 309c116f1e44
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     1
Summary.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     2
--------
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     3
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     4
Testing of a userland component that provides tests is performed by hooking
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     5
those tests up to the 'test' target and running 'gmake test'. This should
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     6
generally pass, as failing tests may indicate things you have to fix, or
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     7
upstream tests that aren't applicable or need modifications on Solaris.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     8
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
     9
'gmake test' is often run when a component is upgraded or otherwise
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    10
intentionally changed, but sometimes it would be useful to rerun the tests
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    11
after something else has changed (such as the system being upgraded, or a
4091
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
    12
change in compilers; see the 'System Test' section below) and see if that
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
    13
has affected the tests.
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    14
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    15
We do this by having a 'master test file' that contains the expected results,
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    16
and having a compare target that runs the tests and compares them with the
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    17
results.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    18
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    19
Note: because the initial test run and the current test run may have a
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    20
different environment (different users, different locales, different machines,
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    21
different compilers...) the results need to have all such output dependencies
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    22
removed or abstracted.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    23
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    24
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    25
Setting up a master test file for a Userland component.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    26
-------------------------------------------------------
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    27
3716
a2629a2cf270 20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3640
diff changeset
    28
When setting up a test-and-compare run for a new component, you will need to
a2629a2cf270 20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3640
diff changeset
    29
have master test file(s) that are identical for both the x86 and SPARC
a2629a2cf270 20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3640
diff changeset
    30
platforms. It is suggested that they should initially be created by doing
a2629a2cf270 20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3640
diff changeset
    31
something like the following in your x86 Userland workspace, then copied to
a2629a2cf270 20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3640
diff changeset
    32
the same locations in your SPARC workspace and retested there.
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    33
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    34
When you run "gmake test", a check is made to see if there is a master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    35
file of test results. If there is, then a test-then-compare run is performed.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    36
If there isn't, then just a "normal" run of the test suite is performed.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    37
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    38
The name of the master test file (or files to be exact), will depend upon
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    39
whether you have 32-bit, 64-bit, 32-and-64-bit and whether this is for a
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    40
Python or Perl component.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    41
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    42
The default master file name is defined in
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    43
make-rules/shared-macros.mk and is:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    44
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    45
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(BITS).master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    46
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    47
where COMPONENT_TEST_RESULTS_DIR is:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    48
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    49
COMPONENT_TEST_RESULTS_DIR =    $(COMPONENT_DIR)/test
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    50
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    51
so that means it will default to looking for the following test file
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    52
master names:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    53
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    54
32-bit: components/<component-name>/test/results-32.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    55
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    56
64-bit: components/<component-name>/test/results-64.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    57
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    58
both:   components/<component-name>/test/results-32.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    59
        components/<component-name>/test/results-64.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    60
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    61
For Python, COMPONENT_TEST_MASTER is overridden in
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    62
make-rules/setup.py.mk to be:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    63
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    64
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PYTHON_VERSION)-$(BITS).master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    65
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    66
so that means it's looking for one or more of:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    67
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    68
2.6:   components/python/<component-name>/test/results-2.6-32.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    69
2.6:   components/python/<component-name>/test/results-2.6-64.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    70
2.7:   components/python/<component-name>/test/results-2.7-32.master
3716
a2629a2cf270 20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3640
diff changeset
    71
2.7:   components/python/<component-name>/test/results-2.7-64.master
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    72
3.4:   components/python/<component-name>/test/results-3.4-64.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    73
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    74
depending upon which versions of Python this component supports.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    75
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    76
Perl is similar, with COMPONENT_TEST_MASTER being overridden in:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    77
make-rules/makemaker.mk to be:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    78
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    79
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PERL_VERSION)-$(BITS).master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    80
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    81
so that means it's looking for one or more of:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    82
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    83
5.12:    components/perl_modules/<component>/test/results-5.12-32.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    84
5.12-mt: components/perl_modules/<component>/test/results-5.12-mt-32.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    85
5.16:    components/perl_modules/<component>/test/results-5.16-64.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    86
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    87
depending upon which versions of Perl this component supports.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    88
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    89
Note that if the test results are the same for both 32-bit and 64-bit or
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    90
for all versions of Python or Perl, you can override the
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    91
COMPONENT_TEST_MASTER definition in your component Makefile and just supply
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    92
a single master files file. For example, in components/python/pep8 Makefile
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    93
we have:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    94
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    95
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-all.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    96
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    97
In order to do a test-then-compare run rather than just run the component
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    98
test suite, initially just create an empty master test file (or files).
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
    99
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   100
For example, for elinks, which just has 64-bit tests, do:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   101
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   102
   $ cd components/elinks
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   103
   $ mkdir test
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   104
   $ touch test/results-64.master
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   105
   $ gmake test
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   106
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   107
At this point, you have a set of test results in
5241
21a30e6ac5fd 22393841 migrate patchutils to Userland
Danek Duvall <danek.duvall@oracle.com>
parents: 4091
diff changeset
   108
components/elinks/build/test/$(MACH64)/test-64-results.
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   109
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   110
Even better, there are a set of "global" regular expressions that are
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   111
run on those test results to try to normalize them. The output from that
5241
21a30e6ac5fd 22393841 migrate patchutils to Userland
Danek Duvall <danek.duvall@oracle.com>
parents: 4091
diff changeset
   112
is placed in components/elinks/build/test/$(MACH64)/results-64.snapshot
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   113
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   114
You can now use the contents of that file as a first cut at the master results.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   115
5241
21a30e6ac5fd 22393841 migrate patchutils to Userland
Danek Duvall <danek.duvall@oracle.com>
parents: 4091
diff changeset
   116
   $ cp build/test/$(MACH64)/results-64.snapshot test/results-64.master
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   117
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   118
Now run the tests again. Note that you have to get back to a clean start
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   119
just in case the test process tries to compile any code.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   120
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   121
   $ gmake clean
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   122
   $ gmake test
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   123
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   124
At this point, it will again compare the test results against the master(s),
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   125
and if there are still differences, they will be placed in
5241
21a30e6ac5fd 22393841 migrate patchutils to Userland
Danek Duvall <danek.duvall@oracle.com>
parents: 4091
diff changeset
   126
components/elinks/build/test/$(MACH64)/test-64-diffs
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   127
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   128
Typically these differences will be for things like usernames, temporary
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   129
filenames, timings etc. If you have some differences, then you are going
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   130
to have to write special regexp expressions in your component Makefile to
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   131
"normalize" them, and adjust the master test results file so that it
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   132
matches what the normalized version looks like.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   133
3744
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   134
Transform examples.
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   135
-------------------
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   136
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   137
1/ Handling temporary filenames.
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   138
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   139
See the transform in the asciidoc Makefile:
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   140
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   141
COMPONENT_TEST_TRANSFORMS += \
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   142
        '-e "s|/tmp......|/tmpxxxxxx|g" '
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   143
3744
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   144
for how to "normalize" any temporary filenames that appear in the test results.
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   145
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   146
2/ Build related output.
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   147
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   148
Items (such as paths to compilers) should be removed from master test files.
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   149
For example, in testing the xz component, we get the following different
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   150
output for compiling a .c file for 32-bit and 64-bit on x86 and SPARC:
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   151
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   152
x86 32-bit:
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   153
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   154
/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I..  -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib  -D_REENTRANT  -m32 -xO4 -xchip=pentium -xregs=no%frameptr    -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   155
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   156
x86 64-bit:
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   157
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   158
/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I..  -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib  -D_REENTRANT  -m64 -xO4 -xchip=generic -Ui386 -U__i386 -xregs=no%frameptr    -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   159
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   160
SPARC 32-bit:
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   161
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   162
/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I..  -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib  -D_REENTRANT  -m32 -xO4 -xtarget=ultra2 -xarch=sparcvis -xchip=ultra2 -Qoption cg -xregs=no%appl -W2,-xwrap_int  -xmemalign=8s -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   163
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   164
SPARC 64-bit:
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   165
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   166
/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I..  -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib  -D_REENTRANT  -m64 -xO4  -xtarget=ultra2 -xarch=sparcvis -xchip=ultra2 -Qoption cg -xregs=no%appl -W2,-xwrap_int  -xmemalign=16s -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   167
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   168
These can all be normalized (so that a single master test file can be used),
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   169
by adding the following line to COMPONENT_TEST_TRANSFORMS in the xz component
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   170
Makefile:
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   171
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   172
	'-e "s|\(^$(CC)\).*|\\$$(CC)|" '
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   173
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   174
The equivalent compile line in the master test file is now set to just:
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   175
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   176
$(CC)
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   177
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   178
Note that we are using $(CC) here which will pick up the current C compiler.
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   179
Should the C compiler or build options change in the future, then there will
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   180
be no need to adjust the master test results file to match. It's all done
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   181
automatically.
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   182
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   183
It is recognized that an alternate approach to solving this problem is to
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   184
create a transform to just remove lines like this from the master test files,
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   185
but there is currently no automatic way to pick up the default compiler
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   186
(see CR #20367119 for more details).
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   187
a74b6fa1af7a 20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents: 3716
diff changeset
   188
----
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   189
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   190
There will be other examples as more components are adjusted to test
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   191
against master results files.
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   192
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   193
If a lot of people start generating the very same ones, then we can
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   194
consider adding them to the "global" set of regexps in
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   195
make-rules/shared-macros.mk which currently looks like:
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   196
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   197
COMPONENT_TEST_TRANSFORMS = \
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   198
        '-e "s|$(@D)|\\$$(@D)|g" ' \
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   199
        '-e "s|$(PERL)|\\$$(PERL)|g" ' \
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   200
        '-e "s|$(SOURCE_DIR)|\\$$(SOURCE_DIR)|g" '
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   201
3747
d2f2bcf116e8 20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents: 3744
diff changeset
   202
Note that in some cases, gsed is not powerful enough.  E.g., if you need to
d2f2bcf116e8 20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents: 3744
diff changeset
   203
do more than massage or remove certain lines, but only compare certain sections
d2f2bcf116e8 20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents: 3744
diff changeset
   204
of the file, then you may need to set COMPONENT_TEST_TRANSFORMER to something
d2f2bcf116e8 20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents: 3744
diff changeset
   205
other than $(GSED) such as $(NAWK).  In this case, you will want to set
d2f2bcf116e8 20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents: 3744
diff changeset
   206
COMPONENT_TEST_TRANSFORMS using "=" instead of "+=", as the default sed
d2f2bcf116e8 20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents: 3744
diff changeset
   207
transforms will likely need to be tossed aside in favor of whatever arguments
d2f2bcf116e8 20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents: 3744
diff changeset
   208
your favorite transformer might need.
d2f2bcf116e8 20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents: 3744
diff changeset
   209
3640
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   210
When your master test file(s) are in good shape, then you should "hg add"
ebe894a8833e 20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff changeset
   211
them to your workspace.
4091
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   212
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   213
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   214
System Test
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   215
-----------
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   216
All of the above discusses how 'gmake test' works.  This is all good, but
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   217
has the limitation that objects under test are those that were just built
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   218
and installed into the component's proto area.  We also want to be able
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   219
to test whatever is installed on the actual system.  This has the benefits
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   220
of not requiring anything to be built, and allowing non-Userland people
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   221
to test our bits.  (Think of people in ON changing libc or the linker and
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   222
wanting to make sure they don't break anything.)
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   223
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   224
To this end, we have added 'gmake system-test'.  It works just like 'gmake
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   225
test', reusing some of the same variables:
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   226
* COMPONENT_TEST_BUILD_DIR
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   227
* COMPONENT_TEST_COMPARE
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   228
* COMPONENT_TEST_CREATE_TRANSFORMS
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   229
* COMPONENT_TEST_OUTPUT
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   230
* COMPONENT_TEST_PERFORM_TRANSFORM
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   231
but with its own "_SYSTEM" instance of other variables:
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   232
* COMPONENT_POST_SYSTEM_TEST_ACTION
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   233
* COMPONENT_PRE_SYSTEM_TEST_ACTION
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   234
* COMPONENT_SYSTEM_TEST_ARGS
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   235
* COMPONENT_SYSTEM_TEST_CLEANUP
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   236
* COMPONENT_SYSTEM_TEST_CMD
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   237
* COMPONENT_SYSTEM_TEST_DIR
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   238
* COMPONENT_SYSTEM_TEST_ENV
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   239
* COMPONENT_SYSTEM_TEST_ENV_CMD
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   240
* COMPONENT_SYSTEM_TEST_RESULTS_DIR
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   241
* COMPONENT_SYSTEM_TEST_TARGETS
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   242
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   243
In the ideal case, only Makefile variables would need to be modified to
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   244
take a component where 'gmake test' works and extend it so that 'gmake
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   245
system-test' works; see components/python34/Makefile for such an example.
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   246
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   247
The next level up from that is the case where some simple patching is needed.
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   248
In this case, simple means:
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   249
* tweaking LD_LIBRARY_PATH
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   250
* changing the path to a binary to run
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   251
* modifying a Makefile so tests are compiled as part of 'gmake build'
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   252
* modifying a Makefile so tests are installed as part of 'gmake install'
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   253
* modifying a pkg(5) manifest so tests are published as part of 'gmake publish'
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   254
  (preferably under the "optional.test" facet)
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   255
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   256
If more extensive patching is needed, a judgement call needs to be made:
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   257
Are these changes that upstream would welcome?  If not, then we would need
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   258
to keep such changes around indefinitely, which would constitute a fork,
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   259
which we try very hard to avoid.  When 'gmake system-test' was initially
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   260
implemented, changes to both fetchmail and python/decorator were begun but
fca9099ab398 20808505 gmake system-test
John Beck <John.Beck@Oracle.COM>
parents: 3747
diff changeset
   261
ultimately abandoned for this reason; see 20808505 for details.