Better Test Code Principles: #2 Don’t generate ALL your test data in @BeforeClass

A considerable amount of tests will need some test data to be generated previous to them running. Some people prefer to put all the data creation for all the tests in a class into the @BeforeClass method, some others prefer to keep the prerequisite data creation inside the tests themselves.

The @BeforeClass mechanism goes something like this:

  • you have a method allocated to this annotation, where you will write code that will be run before any code from the test methods are run
  • if all the code in the @BeforeClass method is run successfully, you will be able to run the tests in the class
  • if any step in the @BeforeClass method throws an exception, no other subsequent code will be run

Here are a few reasons why i would not suggest placing all the prerequisite data generation into the @BeforClass section:

  1. If there is an exception encountered while running any step in the @BeforeClass method, the execution of the @BeforeClass will be stopped.  That means that some of the steps of the @BeforeClass might have been successful, but the first step that will fail will cause the whole method to finish abruptly. Since this step failed, no test from the test class will run. All tests from the class having this @BeforeClass method will be skipped. This means not even tests whose test data generation was successful will not start (those steps that where successful, before the failure was encountered). In this case, all the tests are dependent on the generating of test data for all other tests. Hence you can say that they are not independent, which is not good. You need to be able to run tests separately, but in this case you cannot. This is a situation you want to avoid – either all tests will be able to start, or none will start at all, given where their prerequisite data is generated.
  2. If for some reason you only want to run a single test out of the whole test class, you will still need to wait for the data generation corresponding to all tests in the class. That is because when you run just that one test, the entire @BeforeClass is still run. Hence you might need to wait for a few minutes just to run a test that might take seconds to finish.
  3. If you are working with testng.xml files, that means you want to select only some test methods to be run from your whole test project. Since these tests each require their @BeforeClass to generate their prerequisite data, yet again you will have to wait for all the lines of code from the @BeforeClass method to be executed before any test can be run. Now consider, as an example, that you want to extract 3 methods from 100 classes. That means about 300 methods to be run. Let’s also assume that the @BeforeClass method of each class contains the data generation steps for all 10 method of each class. That means that in order to run 300 tests, you will actually need to wait for the data generation of 1000 tests. And that will take much more time than if you had separated the data generation steps into each test method, as is required.

Having said all the above, it is better to have the test data generation code written inside the test methods that require it. Hence tests will be independent and your @BeforeClass will be smaller and cleaner.

Advertisements

2 thoughts on “Better Test Code Principles: #2 Don’t generate ALL your test data in @BeforeClass”

  1. I agree that moving the test data creation code to the test methods that require it solves the problems that you described (not executing certain tests if there is an error when generating test data, waiting for irrelevant test data to be generated when running only one test, etc.). However, by doing this you are re-introducing the problem of executing an expensive test data generation step multiple times.

    I have an idea for another solution to this problem. Suppose that I have a test class for Foo with four test methods:

    FooTest
    setUpClass -> X, Y
    test1 -> X
    test2 -> X
    test3 -> Y
    test4 -> Y

    In this example, the setUpClass (@BeforeClass) method sets up test data X and Y. Test methods test1 and test2 depend on test data X, and test methods test3 and test4 depend on test data Y. Instead of moving the test data creation to the individual test methods, you could split up the tests into two classes instead:

    FooTest_X
    setUpClass -> X
    test1 -> X
    test2 -> X

    FooTest_Y
    setUpClass -> Y
    test3 -> Y
    test4 -> Y

    Now each test method only needs to wait for relevant test data to be generated before running. This doesn’t work as well for more complex situations (e.g., there is a test method test5 that depends on both test data X and Y — in that case you might want a third test class), but I think that it handles simple situations where each test method only depends on one piece of test data.

    Any thoughts on this idea? Am I missing something?

    Like

    1. Great input! There might indeed arise such situations, as in software there are always exceptions 🙂
      In this case what i would recommend is performing a bit of analysis, as follows:
      – regarding the tests that need the same data as input, you might want to analyse whether you really need more than one test there. An example i can think of, that i’ve seen a lot, is of tests that: only need to open one page on which they will do some checks; they open the page in @BeforeClass; the only thing each test does is check some labels on the page; the checks are grouped in test methods per section on the page (each section having a corresponding test). In this case, it might be useful to just have one test, so the data generation (in this case opening the page) could be moved to the test method.
      – if it is not suitable to group all the checks into one test, than what you propose sounds feasible. You could try moving tests that use the same prerequisite data into a separate class. This brings the advantage that you at least allow for the tests that do not depend on the same prerequisite data to run independently. Thus you minimize the impact of a possible failure in the @BeforeClass to just those tests that really need the data from the @BeforeClass method.
      So basically the whole idea is to try and minimize the dependency of the test methods to the @BeforeClass, as much as possible.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s