A false myth: automated tests don’t uncover bugs

This is something i heard quite a few times: why bother writing automated tests, as they will not find any bugs anyway?

Why do you write automated tests in the first place? Because you want to avoid manually running the same scenario over and over again. You need to test a feature many times: after an update has been made to the feature covered by the test, to run a regression previous to a production release, to gather metrics on the feature, and so on.

So, two ideas to remember: you will test the same feature many times; the application under test which includes the tested feature changes constantly.

Let’s take a simple scenario: a new feature is implemented. Tests need to be written for that feature. While writing the tests you might uncover some bugs, which will be fixed by the development team prior to delivering the feature to the customers. Therefore, by the time the automated test is written, the bugs might already be fixed. This means that at this time, running your automated tests will not find any bugs, since you already found all of them before or during writing the automated tests.

So, when will your automated tests uncover bugs? Here are a few possibilities:

  • when you are using an external library for your feature, but the bits you are using from that library change and are not backwards compatible (those bits of code have a different behavior than expected)
  • when someone performs refactoring on the code that already worked and breaks it
  • when there are issues at branch merges – if people work on several branches, on the same code, and they all merge what they were doing
  • when another part of the application is changing but it is impacting your feature, even though the code of your feature was not changed
  • basic common mistakes of someone altering the code unwillingly
  • changes to underlying hardware equipment might also cause bugs; they might not be functional ones, but performance degradation might be observed and captured by running the automated tests

Furthermore, it is advisable that after a new bug is found in a scenario that was not covered by automated tests, a new one should be written. Just to make sure that if the bug reappears, you will be aware of it. Of course, if it is worth it from a time and effort perspective, and if the feature is deemed important or major.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s