run junit tests in a certain order

[J. Xue]

I'm sure I'm not the first one having this problem but can't find anything on
google.

I would like to be able to run some JUnit tests in a certain order, e.g.,
starting from the most basic ones. There doesn't seem to be a perfect way to
do this using the <junit> task. <batchtest> takes a fileset, so the order of
the test files is basically determined by File.listFile(), which returns files
in an OS-specific order.

For a workaround, I wrote a master test suite, in which I read from a file
that has the list of test files in the order I want them to run. I then run
it using <test>. Now everything works fine as far as order enforcement is
concerned - but all the test methods are listed under the master test suite
class in the report! So if I have two test methods from different classes
with the same name, I wouldn't be able to tell which is which.

Other options include hardcoding the order into build.xml, or hack the source
code for <batchtest> to take a file list. Neither of those paths smells like
a pleasant solution, though.

Any hint would be appreciated.
--J.X.

[Martin Gainty]

Derive your own class from File
code the listFile() method ordering the way you see fit..
-Martin
----- Original Message -----
From: "J. Xue" <***@manifoldronin.com>
To: "Ant Users List" <***@ant.apache.org>
Sent: Tuesday, November 25, 2003 7:49 PM
Subject: run junit tests in a certain order

Post by J. Xue
I'm sure I'm not the first one having this problem but can't find anything on
google.
I would like to be able to run some JUnit tests in a certain order, e.g.,
starting from the most basic ones. There doesn't seem to be a perfect way to
do this using the <junit> task. <batchtest> takes a fileset, so the order of
the test files is basically determined by File.listFile(), which returns files
in an OS-specific order.
For a workaround, I wrote a master test suite, in which I read from a file
that has the list of test files in the order I want them to run. I then run
it using <test>. Now everything works fine as far as order enforcement is
concerned - but all the test methods are listed under the master test suite
class in the report! So if I have two test methods from different classes
with the same name, I wouldn't be able to tell which is which.
Other options include hardcoding the order into build.xml, or hack the source
code for <batchtest> to take a file list. Neither of those paths smells like
a pleasant solution, though.
Any hint would be appreciated.
--J.X.
---------------------------------------------------------------------

[J. Xue]

Post by Martin Gainty
Derive your own class from File
code the listFile() method ordering the way you see fit..
-Martin

Either we are not talking about the same good old java.io.File, or I need a
second cup of coffee this morning - How do I make Ant use my subclass instead
of File?

--J.X.

[Stefan Bodewig]

Post by J. Xue
I'm sure I'm not the first one having this problem but can't find
anything on google.

Well, then you didn't use the right keywords 8-)

Generally, JUnit advocates are going to tell you that you are trying
to do the wrong thing in the first place as unit tests are supposed to
be independent of each other.

Now to your problem:

An alternative to your approach is to explicitly use <test> elements
for the tests that need a certain order and exclude them from the
<batchtest> that contains the rest.

Post by J. Xue
For a workaround, I wrote a master test suite, in which I read from
a file that has the list of test files in the order I want them to
run. I then run it using <test>. Now everything works fine as far
as order enforcement is concerned - but all the test methods are
listed under the master test suite class in the report!

With Ant 1.6beta, all necessary information is inside the XML reports,
but I don't think that <junitreport> uses it right now (an attribute
has been added to the test elements). If you use the beta and modify
the XSLs to use the new attribute, your reports should work as well.

Stefan

[J. Xue]

Post by Stefan Bodewig

Post by J. Xue
I'm sure I'm not the first one having this problem but can't find
anything on google.

Well, then you didn't use the right keywords 8-)

Well, figuring out the right google keywords has been one of the hardest parts
in my computing experience. 8-)

Post by Stefan Bodewig
Generally, JUnit advocates are going to tell you that you are trying
to do the wrong thing in the first place as unit tests are supposed to
be independent of each other.

I totally agree, but then it doesn't contradict to what I'm trying to do. My
tests are independent. I just feel it a bit silly to waste 20 mins to run a
AdvancedTests suite that's doomed to fail because the SanityCheckTests suite
would have failed quickly, had it not been because of the OS I'm running on
happens to decide to return files in alphabetical order.

Oh well, this ain't the junit list... 8-)

Post by Stefan Bodewig
An alternative to your approach is to explicitly use <test> elements
for the tests that need a certain order and exclude them from the
<batchtest> that contains the rest.

I do realize I have this option, but then I'd have to hardcode the test order
in build.xml. Another disadvantage is that since I always fork the runner VM,
having a lot of <test>/<batchtest> elements would tend to slow down the build.

Post by Stefan Bodewig

Post by J. Xue
For a workaround, I wrote a master test suite, in which I read from
a file that has the list of test files in the order I want them to
run. I then run it using <test>. Now everything works fine as far
as order enforcement is concerned - but all the test methods are
listed under the master test suite class in the report!

With Ant 1.6beta, all necessary information is inside the XML reports,
but I don't think that <junitreport> uses it right now (an attribute
has been added to the test elements). If you use the beta and modify
the XSLs to use the new attribute, your reports should work as well.

I'll definitely give it a try tonight. Thanks.
--J.X.

[Wascally Wabbit]

Post by J. Xue

Post by Stefan Bodewig

Post by J. Xue
I'm sure I'm not the first one having this problem but can't find
anything on google.

Well, then you didn't use the right keywords 8-)

Well, figuring out the right google keywords has been one of the hardest parts
in my computing experience. 8-)

Post by Stefan Bodewig
Generally, JUnit advocates are going to tell you that you are trying
to do the wrong thing in the first place as unit tests are supposed to
be independent of each other.

I totally agree, but then it doesn't contradict to what I'm trying to do. My
tests are independent. I just feel it a bit silly to waste 20 mins to run a
AdvancedTests suite that's doomed to fail because the SanityCheckTests suite
would have failed quickly, had it not been because of the OS I'm running on
happens to decide to return files in alphabetical order.

We have a similiar problem with large (long running) test suites. Our
solution was to create a "BaselineSuite" in addition to the JUnit advised
"AllTests" where this sort of smoke-testing is useful. The baseline
suite files are named (surprise) *BaslineSuite.java. We then
changed our standard "run acceptance tests" targets to first do baseline
suites, if all these pass then we do the full long-winded suites.

The downside is now you have to manually maintain the Baseline as
well as the AllTests suite (although creating baselines turned out to
be a useful exercise for qa/review folks).


The Wabbit

[Stefan Bodewig]

Post by J. Xue

Post by Stefan Bodewig
An alternative to your approach is to explicitly use <test>
elements for the tests that need a certain order and exclude them
from the <batchtest> that contains the rest.

I do realize I have this option, but then I'd have to hardcode the
test order in build.xml.

But only for the few files that would fail quickly - or alternatively
for the few files that run really really long (put the single <test>s
after the <batchtest>).

Post by J. Xue
Another disadvantage is that since I always fork the runner VM,
having a lot of <test>/<batchtest> elements would tend to slow down the build.

Absolutely, yes.

I'll try to make that one forked VM for all tests in 1.7 (as an
option), but we are not there yet.

Stefan

[Laurent Duperval]

Post by Stefan Bodewig

Post by J. Xue

Post by Stefan Bodewig
An alternative to your approach is to explicitly use <test>
elements for the tests that need a certain order and exclude them
from the <batchtest> that contains the rest.

I do realize I have this option, but then I'd have to hardcode the
test order in build.xml.

But only for the few files that would fail quickly - or alternatively
for the few files that run really really long (put the single <test>s
after the <batchtest>).

You could also have different class names for the tests you want to run
first. Then your test target would be comprised of two depends: one for
thequick tests and the other for everything else.

Not ideal.

L