|
1 Summary. |
|
2 -------- |
|
3 |
|
4 Testing of a userland component that provides tests is performed by hooking |
|
5 those tests up to the 'test' target and running 'gmake test'. This should |
|
6 generally pass, as failing tests may indicate things you have to fix, or |
|
7 upstream tests that aren't applicable or need modifications on Solaris. |
|
8 |
|
9 'gmake test' is often run when a component is upgraded or otherwise |
|
10 intentionally changed, but sometimes it would be useful to rerun the tests |
|
11 after something else has changed (such as the system being upgraded, or a |
|
12 change in compilers) and see if that has affected the tests. |
|
13 |
|
14 We do this by having a 'master test file' that contains the expected results, |
|
15 and having a compare target that runs the tests and compares them with the |
|
16 results. |
|
17 |
|
18 Note: because the initial test run and the current test run may have a |
|
19 different environment (different users, different locales, different machines, |
|
20 different compilers...) the results need to have all such output dependencies |
|
21 removed or abstracted. |
|
22 |
|
23 |
|
24 Setting up a master test file for a Userland component. |
|
25 ------------------------------------------------------- |
|
26 |
|
27 When setting up a test-and-compare run for a new component, |
|
28 you should be able to do something like the following. |
|
29 |
|
30 When you run "gmake test", a check is made to see if there is a master |
|
31 file of test results. If there is, then a test-then-compare run is performed. |
|
32 If there isn't, then just a "normal" run of the test suite is performed. |
|
33 |
|
34 The name of the master test file (or files to be exact), will depend upon |
|
35 whether you have 32-bit, 64-bit, 32-and-64-bit and whether this is for a |
|
36 Python or Perl component. |
|
37 |
|
38 The default master file name is defined in |
|
39 make-rules/shared-macros.mk and is: |
|
40 |
|
41 COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(BITS).master |
|
42 |
|
43 where COMPONENT_TEST_RESULTS_DIR is: |
|
44 |
|
45 COMPONENT_TEST_RESULTS_DIR = $(COMPONENT_DIR)/test |
|
46 |
|
47 so that means it will default to looking for the following test file |
|
48 master names: |
|
49 |
|
50 32-bit: components/<component-name>/test/results-32.master |
|
51 |
|
52 64-bit: components/<component-name>/test/results-64.master |
|
53 |
|
54 both: components/<component-name>/test/results-32.master |
|
55 components/<component-name>/test/results-64.master |
|
56 |
|
57 For Python, COMPONENT_TEST_MASTER is overridden in |
|
58 make-rules/setup.py.mk to be: |
|
59 |
|
60 COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PYTHON_VERSION)-$(BITS).master |
|
61 |
|
62 so that means it's looking for one or more of: |
|
63 |
|
64 2.6: components/python/<component-name>/test/results-2.6-32.master |
|
65 2.6: components/python/<component-name>/test/results-2.6-64.master |
|
66 2.7: components/python/<component-name>/test/results-2.7-32.master |
|
67 2.7: components/python/<component-name>/test/results-3.4-64.master |
|
68 3.4: components/python/<component-name>/test/results-3.4-64.master |
|
69 |
|
70 depending upon which versions of Python this component supports. |
|
71 |
|
72 Perl is similar, with COMPONENT_TEST_MASTER being overridden in: |
|
73 make-rules/makemaker.mk to be: |
|
74 |
|
75 COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PERL_VERSION)-$(BITS).master |
|
76 |
|
77 so that means it's looking for one or more of: |
|
78 |
|
79 5.12: components/perl_modules/<component>/test/results-5.12-32.master |
|
80 5.12-mt: components/perl_modules/<component>/test/results-5.12-mt-32.master |
|
81 5.16: components/perl_modules/<component>/test/results-5.16-64.master |
|
82 |
|
83 depending upon which versions of Perl this component supports. |
|
84 |
|
85 Note that if the test results are the same for both 32-bit and 64-bit or |
|
86 for all versions of Python or Perl, you can override the |
|
87 COMPONENT_TEST_MASTER definition in your component Makefile and just supply |
|
88 a single master files file. For example, in components/python/pep8 Makefile |
|
89 we have: |
|
90 |
|
91 COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-all.master |
|
92 |
|
93 In order to do a test-then-compare run rather than just run the component |
|
94 test suite, initially just create an empty master test file (or files). |
|
95 |
|
96 For example, for elinks, which just has 64-bit tests, do: |
|
97 |
|
98 $ cd components/elinks |
|
99 $ mkdir test |
|
100 $ touch test/results-64.master |
|
101 $ gmake test |
|
102 |
|
103 At this point, you have a set of test results in |
|
104 components/elinks/test/test-64-results. |
|
105 |
|
106 Even better, there are a set of "global" regular expressions that are |
|
107 run on those test results to try to normalize them. The output from that |
|
108 is placed in components/elinks/test/results-64.snapshot |
|
109 |
|
110 You can now use the contents of that file as a first cut at the master results. |
|
111 |
|
112 $ cp test/results-64.snapshot test/results-64.master |
|
113 |
|
114 Now run the tests again. Note that you have to get back to a clean start |
|
115 just in case the test process tries to compile any code. |
|
116 |
|
117 $ gmake clean |
|
118 $ gmake test |
|
119 |
|
120 At this point, it will again compare the test results against the master(s), |
|
121 and if there are still differences, they will be placed in |
|
122 components/elinks/test/test-64-diffs |
|
123 |
|
124 Typically these differences will be for things like usernames, temporary |
|
125 filenames, timings etc. If you have some differences, then you are going |
|
126 to have to write special regexp expressions in your component Makefile to |
|
127 "normalize" them, and adjust the master test results file so that it |
|
128 matches what the normalized version looks like. |
|
129 |
|
130 For example, see the transform in the asciidoc Makefile: |
|
131 |
|
132 COMPONENT_TEST_TRANSFORMS += \ |
|
133 '-e "s|/tmp......|/tmpxxxxxx|g" ' |
|
134 |
|
135 to "normalize" any temporary filenames that appear in the test results. |
|
136 |
|
137 There will be other examples as more components are adjusted to test |
|
138 against master results files. |
|
139 |
|
140 If a lot of people start generating the very same ones, then we can |
|
141 consider adding them to the "global" set of regexps in |
|
142 make-rules/shared-macros.mk which currently looks like: |
|
143 |
|
144 COMPONENT_TEST_TRANSFORMS = \ |
|
145 '-e "s|$(@D)|\\$$(@D)|g" ' \ |
|
146 '-e "s|$(PERL)|\\$$(PERL)|g" ' \ |
|
147 '-e "s|$(SOURCE_DIR)|\\$$(SOURCE_DIR)|g" ' |
|
148 |
|
149 When your master test file(s) are in good shape, then you should "hg add" |
|
150 them to your workspace. |