author | John Beck <John.Beck@Oracle.COM> |
Mon, 02 Mar 2015 16:15:04 -0800 | |
changeset 3898 | acd7f120a52a |
parent 3747 | d2f2bcf116e8 |
child 4091 | fca9099ab398 |
permissions | -rw-r--r-- |
3640
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
1 |
Summary. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
2 |
-------- |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
3 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
4 |
Testing of a userland component that provides tests is performed by hooking |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
5 |
those tests up to the 'test' target and running 'gmake test'. This should |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
6 |
generally pass, as failing tests may indicate things you have to fix, or |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
7 |
upstream tests that aren't applicable or need modifications on Solaris. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
8 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
9 |
'gmake test' is often run when a component is upgraded or otherwise |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
10 |
intentionally changed, but sometimes it would be useful to rerun the tests |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
11 |
after something else has changed (such as the system being upgraded, or a |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
12 |
change in compilers) and see if that has affected the tests. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
13 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
14 |
We do this by having a 'master test file' that contains the expected results, |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
15 |
and having a compare target that runs the tests and compares them with the |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
16 |
results. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
17 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
18 |
Note: because the initial test run and the current test run may have a |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
19 |
different environment (different users, different locales, different machines, |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
20 |
different compilers...) the results need to have all such output dependencies |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
21 |
removed or abstracted. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
22 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
23 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
24 |
Setting up a master test file for a Userland component. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
25 |
------------------------------------------------------- |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
26 |
|
3716
a2629a2cf270
20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3640
diff
changeset
|
27 |
When setting up a test-and-compare run for a new component, you will need to |
a2629a2cf270
20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3640
diff
changeset
|
28 |
have master test file(s) that are identical for both the x86 and SPARC |
a2629a2cf270
20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3640
diff
changeset
|
29 |
platforms. It is suggested that they should initially be created by doing |
a2629a2cf270
20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3640
diff
changeset
|
30 |
something like the following in your x86 Userland workspace, then copied to |
a2629a2cf270
20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3640
diff
changeset
|
31 |
the same locations in your SPARC workspace and retested there. |
3640
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
32 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
33 |
When you run "gmake test", a check is made to see if there is a master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
34 |
file of test results. If there is, then a test-then-compare run is performed. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
35 |
If there isn't, then just a "normal" run of the test suite is performed. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
36 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
37 |
The name of the master test file (or files to be exact), will depend upon |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
38 |
whether you have 32-bit, 64-bit, 32-and-64-bit and whether this is for a |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
39 |
Python or Perl component. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
40 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
41 |
The default master file name is defined in |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
42 |
make-rules/shared-macros.mk and is: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
43 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
44 |
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(BITS).master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
45 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
46 |
where COMPONENT_TEST_RESULTS_DIR is: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
47 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
48 |
COMPONENT_TEST_RESULTS_DIR = $(COMPONENT_DIR)/test |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
49 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
50 |
so that means it will default to looking for the following test file |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
51 |
master names: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
52 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
53 |
32-bit: components/<component-name>/test/results-32.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
54 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
55 |
64-bit: components/<component-name>/test/results-64.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
56 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
57 |
both: components/<component-name>/test/results-32.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
58 |
components/<component-name>/test/results-64.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
59 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
60 |
For Python, COMPONENT_TEST_MASTER is overridden in |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
61 |
make-rules/setup.py.mk to be: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
62 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
63 |
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PYTHON_VERSION)-$(BITS).master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
64 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
65 |
so that means it's looking for one or more of: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
66 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
67 |
2.6: components/python/<component-name>/test/results-2.6-32.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
68 |
2.6: components/python/<component-name>/test/results-2.6-64.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
69 |
2.7: components/python/<component-name>/test/results-2.7-32.master |
3716
a2629a2cf270
20451511 Yet more Userland components should have master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3640
diff
changeset
|
70 |
2.7: components/python/<component-name>/test/results-2.7-64.master |
3640
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
71 |
3.4: components/python/<component-name>/test/results-3.4-64.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
72 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
73 |
depending upon which versions of Python this component supports. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
74 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
75 |
Perl is similar, with COMPONENT_TEST_MASTER being overridden in: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
76 |
make-rules/makemaker.mk to be: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
77 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
78 |
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PERL_VERSION)-$(BITS).master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
79 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
80 |
so that means it's looking for one or more of: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
81 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
82 |
5.12: components/perl_modules/<component>/test/results-5.12-32.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
83 |
5.12-mt: components/perl_modules/<component>/test/results-5.12-mt-32.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
84 |
5.16: components/perl_modules/<component>/test/results-5.16-64.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
85 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
86 |
depending upon which versions of Perl this component supports. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
87 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
88 |
Note that if the test results are the same for both 32-bit and 64-bit or |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
89 |
for all versions of Python or Perl, you can override the |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
90 |
COMPONENT_TEST_MASTER definition in your component Makefile and just supply |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
91 |
a single master files file. For example, in components/python/pep8 Makefile |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
92 |
we have: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
93 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
94 |
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-all.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
95 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
96 |
In order to do a test-then-compare run rather than just run the component |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
97 |
test suite, initially just create an empty master test file (or files). |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
98 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
99 |
For example, for elinks, which just has 64-bit tests, do: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
100 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
101 |
$ cd components/elinks |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
102 |
$ mkdir test |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
103 |
$ touch test/results-64.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
104 |
$ gmake test |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
105 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
106 |
At this point, you have a set of test results in |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
107 |
components/elinks/test/test-64-results. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
108 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
109 |
Even better, there are a set of "global" regular expressions that are |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
110 |
run on those test results to try to normalize them. The output from that |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
111 |
is placed in components/elinks/test/results-64.snapshot |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
112 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
113 |
You can now use the contents of that file as a first cut at the master results. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
114 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
115 |
$ cp test/results-64.snapshot test/results-64.master |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
116 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
117 |
Now run the tests again. Note that you have to get back to a clean start |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
118 |
just in case the test process tries to compile any code. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
119 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
120 |
$ gmake clean |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
121 |
$ gmake test |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
122 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
123 |
At this point, it will again compare the test results against the master(s), |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
124 |
and if there are still differences, they will be placed in |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
125 |
components/elinks/test/test-64-diffs |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
126 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
127 |
Typically these differences will be for things like usernames, temporary |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
128 |
filenames, timings etc. If you have some differences, then you are going |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
129 |
to have to write special regexp expressions in your component Makefile to |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
130 |
"normalize" them, and adjust the master test results file so that it |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
131 |
matches what the normalized version looks like. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
132 |
|
3744
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
133 |
Transform examples. |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
134 |
------------------- |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
135 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
136 |
1/ Handling temporary filenames. |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
137 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
138 |
See the transform in the asciidoc Makefile: |
3640
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
139 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
140 |
COMPONENT_TEST_TRANSFORMS += \ |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
141 |
'-e "s|/tmp......|/tmpxxxxxx|g" ' |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
142 |
|
3744
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
143 |
for how to "normalize" any temporary filenames that appear in the test results. |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
144 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
145 |
2/ Build related output. |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
146 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
147 |
Items (such as paths to compilers) should be removed from master test files. |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
148 |
For example, in testing the xz component, we get the following different |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
149 |
output for compiling a .c file for 32-bit and 64-bit on x86 and SPARC: |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
150 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
151 |
x86 32-bit: |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
152 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
153 |
/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I.. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib -D_REENTRANT -m32 -xO4 -xchip=pentium -xregs=no%frameptr -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
154 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
155 |
x86 64-bit: |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
156 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
157 |
/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I.. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib -D_REENTRANT -m64 -xO4 -xchip=generic -Ui386 -U__i386 -xregs=no%frameptr -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
158 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
159 |
SPARC 32-bit: |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
160 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
161 |
/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I.. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib -D_REENTRANT -m32 -xO4 -xtarget=ultra2 -xarch=sparcvis -xchip=ultra2 -Qoption cg -xregs=no%appl -W2,-xwrap_int -xmemalign=8s -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
162 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
163 |
SPARC 64-bit: |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
164 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
165 |
/ws/on12-tools/SUNWspro/solarisstudio12.3/bin/cc -DHAVE_CONFIG_H -I. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/tests -I.. -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/common -I/builds/richb/twisted-tests/components/xz/xz-5.0.1/src/liblzma/api -I../lib -D_REENTRANT -m64 -xO4 -xtarget=ultra2 -xarch=sparcvis -xchip=ultra2 -Qoption cg -xregs=no%appl -W2,-xwrap_int -xmemalign=16s -mt -c /builds/richb/twisted-tests/components/xz/xz-5.0.1/tests/test_check.c |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
166 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
167 |
These can all be normalized (so that a single master test file can be used), |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
168 |
by adding the following line to COMPONENT_TEST_TRANSFORMS in the xz component |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
169 |
Makefile: |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
170 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
171 |
'-e "s|\(^$(CC)\).*|\\$$(CC)|" ' |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
172 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
173 |
The equivalent compile line in the master test file is now set to just: |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
174 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
175 |
$(CC) |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
176 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
177 |
Note that we are using $(CC) here which will pick up the current C compiler. |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
178 |
Should the C compiler or build options change in the future, then there will |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
179 |
be no need to adjust the master test results file to match. It's all done |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
180 |
automatically. |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
181 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
182 |
It is recognized that an alternate approach to solving this problem is to |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
183 |
create a transform to just remove lines like this from the master test files, |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
184 |
but there is currently no automatic way to pick up the default compiler |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
185 |
(see CR #20367119 for more details). |
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
186 |
|
a74b6fa1af7a
20455804 TWISTED should have some master test results to compare against
Rich Burridge <rich.burridge@oracle.com>
parents:
3716
diff
changeset
|
187 |
---- |
3640
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
188 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
189 |
There will be other examples as more components are adjusted to test |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
190 |
against master results files. |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
191 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
192 |
If a lot of people start generating the very same ones, then we can |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
193 |
consider adding them to the "global" set of regexps in |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
194 |
make-rules/shared-macros.mk which currently looks like: |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
195 |
|
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
196 |
COMPONENT_TEST_TRANSFORMS = \ |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
197 |
'-e "s|$(@D)|\\$$(@D)|g" ' \ |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
198 |
'-e "s|$(PERL)|\\$$(PERL)|g" ' \ |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
199 |
'-e "s|$(SOURCE_DIR)|\\$$(SOURCE_DIR)|g" ' |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
200 |
|
3747
d2f2bcf116e8
20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents:
3744
diff
changeset
|
201 |
Note that in some cases, gsed is not powerful enough. E.g., if you need to |
d2f2bcf116e8
20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents:
3744
diff
changeset
|
202 |
do more than massage or remove certain lines, but only compare certain sections |
d2f2bcf116e8
20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents:
3744
diff
changeset
|
203 |
of the file, then you may need to set COMPONENT_TEST_TRANSFORMER to something |
d2f2bcf116e8
20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents:
3744
diff
changeset
|
204 |
other than $(GSED) such as $(NAWK). In this case, you will want to set |
d2f2bcf116e8
20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents:
3744
diff
changeset
|
205 |
COMPONENT_TEST_TRANSFORMS using "=" instead of "+=", as the default sed |
d2f2bcf116e8
20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents:
3744
diff
changeset
|
206 |
transforms will likely need to be tossed aside in favor of whatever arguments |
d2f2bcf116e8
20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents:
3744
diff
changeset
|
207 |
your favorite transformer might need. |
d2f2bcf116e8
20441004 Python master test results
John Beck <John.Beck@Oracle.COM>
parents:
3744
diff
changeset
|
208 |
|
3640
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
209 |
When your master test file(s) are in good shape, then you should "hg add" |
ebe894a8833e
20359322 Need documentation for adding master test results to Userland components
Rich Burridge <rich.burridge@oracle.com>
parents:
diff
changeset
|
210 |
them to your workspace. |