7 upstream tests that aren't applicable or need modifications on Solaris. |
7 upstream tests that aren't applicable or need modifications on Solaris. |
8 |
8 |
9 'gmake test' is often run when a component is upgraded or otherwise |
9 'gmake test' is often run when a component is upgraded or otherwise |
10 intentionally changed, but sometimes it would be useful to rerun the tests |
10 intentionally changed, but sometimes it would be useful to rerun the tests |
11 after something else has changed (such as the system being upgraded, or a |
11 after something else has changed (such as the system being upgraded, or a |
12 change in compilers) and see if that has affected the tests. |
12 change in compilers; see the 'System Test' section below) and see if that |
|
13 has affected the tests. |
13 |
14 |
14 We do this by having a 'master test file' that contains the expected results, |
15 We do this by having a 'master test file' that contains the expected results, |
15 and having a compare target that runs the tests and compares them with the |
16 and having a compare target that runs the tests and compares them with the |
16 results. |
17 results. |
17 |
18 |
206 transforms will likely need to be tossed aside in favor of whatever arguments |
207 transforms will likely need to be tossed aside in favor of whatever arguments |
207 your favorite transformer might need. |
208 your favorite transformer might need. |
208 |
209 |
209 When your master test file(s) are in good shape, then you should "hg add" |
210 When your master test file(s) are in good shape, then you should "hg add" |
210 them to your workspace. |
211 them to your workspace. |
|
212 |
|
213 |
|
214 System Test |
|
215 ----------- |
|
216 All of the above discusses how 'gmake test' works. This is all good, but |
|
217 has the limitation that objects under test are those that were just built |
|
218 and installed into the component's proto area. We also want to be able |
|
219 to test whatever is installed on the actual system. This has the benefits |
|
220 of not requiring anything to be built, and allowing non-Userland people |
|
221 to test our bits. (Think of people in ON changing libc or the linker and |
|
222 wanting to make sure they don't break anything.) |
|
223 |
|
224 To this end, we have added 'gmake system-test'. It works just like 'gmake |
|
225 test', reusing some of the same variables: |
|
226 * COMPONENT_TEST_BUILD_DIR |
|
227 * COMPONENT_TEST_COMPARE |
|
228 * COMPONENT_TEST_CREATE_TRANSFORMS |
|
229 * COMPONENT_TEST_OUTPUT |
|
230 * COMPONENT_TEST_PERFORM_TRANSFORM |
|
231 but with its own "_SYSTEM" instance of other variables: |
|
232 * COMPONENT_POST_SYSTEM_TEST_ACTION |
|
233 * COMPONENT_PRE_SYSTEM_TEST_ACTION |
|
234 * COMPONENT_SYSTEM_TEST_ARGS |
|
235 * COMPONENT_SYSTEM_TEST_CLEANUP |
|
236 * COMPONENT_SYSTEM_TEST_CMD |
|
237 * COMPONENT_SYSTEM_TEST_DIR |
|
238 * COMPONENT_SYSTEM_TEST_ENV |
|
239 * COMPONENT_SYSTEM_TEST_ENV_CMD |
|
240 * COMPONENT_SYSTEM_TEST_RESULTS_DIR |
|
241 * COMPONENT_SYSTEM_TEST_TARGETS |
|
242 |
|
243 In the ideal case, only Makefile variables would need to be modified to |
|
244 take a component where 'gmake test' works and extend it so that 'gmake |
|
245 system-test' works; see components/python34/Makefile for such an example. |
|
246 |
|
247 The next level up from that is the case where some simple patching is needed. |
|
248 In this case, simple means: |
|
249 * tweaking LD_LIBRARY_PATH |
|
250 * changing the path to a binary to run |
|
251 * modifying a Makefile so tests are compiled as part of 'gmake build' |
|
252 * modifying a Makefile so tests are installed as part of 'gmake install' |
|
253 * modifying a pkg(5) manifest so tests are published as part of 'gmake publish' |
|
254 (preferably under the "optional.test" facet) |
|
255 |
|
256 If more extensive patching is needed, a judgement call needs to be made: |
|
257 Are these changes that upstream would welcome? If not, then we would need |
|
258 to keep such changes around indefinitely, which would constitute a fork, |
|
259 which we try very hard to avoid. When 'gmake system-test' was initially |
|
260 implemented, changes to both fetchmail and python/decorator were begun but |
|
261 ultimately abandoned for this reason; see 20808505 for details. |