What are Exclude and Include Test Methods in TestNG?

Test Driven Development using JUnit5 and Mockito

TestNG is a popular Java testing framework that stands for Test-Next Generation. It is an advanced form of JUnit, and offers comprehensive features, especially when it comes to integration tests. In this article, we will learn how to include and exclude test cases from executing at the Method, Class, and Suite levels.

Table of Content

Include Test Methods
Exclude Test Methods
Advanced Techniques for Exclude and Include
Conclusion
FAQ’s on Exclude Include Test Cases in TestNG

Include and exclude features in TestNG allow us to stop and allow tests from executing on the go. This means that if we have written a test that we do not want to execute in some situation, we can simply ignore that test case, test method, or test suite without having to delete it entirely. This way, if we ever need to run these tests again, all we need to do is simply enable them.

Include Test Methods

@Test annotation

The @Test annotation marks a method as a unit test in TestNG. It tells the TestNG execution environment to include the following method in the test execution as a unit test.

@Testpublic void LoginTest() { //test body here}

The @Test annotation has several properties that can be used to execute it as we want. The three most important properties of @Test annotation are:

enabled: This property is like a switch. It turns the method on or off. If you set enabled=false, the test method will be stopped from execution and act like it does not exist.
dependsOnMethods: This property allows you to specify test dependency and control the order of Tests. It takes an array of Strings as arguments. The strings are the names of the methods the following method depends on. The execution of these must be complete before we try to run the succeeding method.
groups: This property is a string array that contains the names of the groups the particular method belongs to. Groups are specified as follows:

@Test(groups={“A”,”B”}, enabled=true)public void LoginTest() { //test body here}

The above example portrays that the method LoginTest belongs to the groups ‘A’ and ‘B’. You can include a method in as many groups as you want. In the next section, we will see how we can run methods in a specific group using the testng.xml file.

Demonstration of including specific test methods using group attribute

Once you have created groups using the above method, you can simply include or exclude the groups you want to run during the execution of test methods in a test suite.

To include/exclude particular groups first create a simple testng.xml file:

XML

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Regression Test Suite">
    <test name="Search Test">
        <classes>
            <class name="com.w3wiki.browsertests.TestClassA" />
            <class name="com.w3wiki.browsertests.TestClassB" />
        </classes>
    </test>
</suite>

To run groups, we will add the <groups> tag to this testng.xml file. The ‘groups’ tag can be added directly within the suite tag as:

XML

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Regression Test Suite">
    <groups>
          <run>......</run>
    </groups>
    <test name="Search Test">
        <classes>....</classes>
    </test>
</suite>

This means that the group tag is universal to the suite. It will apply to all test tags within that particular testng.xml file. Alternatively, you can also use this tag inside the test tag as:

XML

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Regression Test Suite">
    <groups>
          <run>......</run>
    </groups>
    <test name="Search Test">
        <groups>
              <run>......</run>
        </groups>
        <classes>....</classes>
    </test>
</suite>

In this scenario, the groups tag will only be applied to the specific test it is included within.

To include/exclude methods using groups, you have to specify the group names in the <run /> tag within the groups tag as:

XML

<groups>
    <run>
        <include name="name of the group to be included" />
        <exclude name="name of the group to be excluded" />
    </run>
</groups>

You can use as many include/exclude tags within the run tag as you need to.

Let’s look at a real example showcasing this functionality.

Example code snippet illustrating inclusion of tests based on group names

Step 1: Setup TestNG Environment

TestNG does not require any special kind of setup incase of IntelliJ Idea, both enterprise and community edition. Incase you choose to follow this tutorial on eclipse IDE, then you need to setup the TestNG environment before you can start using it in your application. To learn how to install and configure the plugin in the Eclipse IDE, please visit this page.

To Create a new TestNG project regardless of the IDE you choose to use, simply – Create a new Maven project and add the TestNG dependencies to it. Maven automatically provides us with a default project structure with a separate place to our unit tests in.

Step 2: Create a new Maven project.

To learn how to create a new maven project, visit this page.

Step 3: Add The following Dependencies to Your pom.xml file

In order to follow the tutorial, please add the selenium dependency for web browser automation and the TestNG maven plugin to execute tests based on configurations in the testng.xml file as follows:

XML

    <dependencies>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>4.18.1</version>
        </dependency>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-server</artifactId>
            <version>4.0.0-alpha-2</version>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.9.0</version>
        </dependency>
        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-core</artifactId>
            <version>2.2</version>
        </dependency>
    </dependencies>

Step 4. Creating Test Classes

Explanation of test classes in TestNG:

Knowledge of TestNG is a basic pre-requisite of this tutorial. Please read this article for some background knowledge. Basically what we do is that we create our tests under the test directory that Maven created for us. Go to test > java > inverse domain name folder > package > classes.
Inverse Domain name means that suppose we have a domain name called w3wiki.net. The inverse domain name of this would be org.w3wiki. This is essential for TestNG to be able to locate your test classes.
Inside the folder with the inverse domain name that we have created, we can have as many packages we need. These test packages can be based on features or they may be based upon the different types of tests. This depends upon the specific needs of the project that you are working on and may be different for different companies.
And lastly, inside these test packages, we can write our test classes. These test classes allow us to write unit tests inside them. We can have as many test classes we need and as many tests inside them as we want.

Step 5. For this tutorial, we will create two test classes called TestClassA and TestClassB inside the java folder in the test directory.

TestClassA:

Java

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import java.time.Duration;

public class TestClassA {
    private WebDriver driver;
    private WebElement searchBar;
    @BeforeClass
    public void setUp() {
        driver=new ChromeDriver();
        driver.get("https://www.w3wiki.net/");
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(3));
        searchBar=driver.findElement(By.xpath("//*[@id=\"RA-root\"]/div/div[1]/div[1]/div[2]/span/span/span[1]/input"));
    }

    @Test(groups = {"page-search"})
    public void testWebPageSearch() {
        searchBar.sendKeys("TestNG and Selenium");
        driver.findElement(By.xpath("//*[@id=\"RA-root\"]/div/div[1]/div[1]/div[2]/span/span/span[2]/button")).click();
    }

    @AfterClass
    public void tearDown() {
        searchBar=null;
        driver=null;
    }
}

TestClassB:

Java

import org.openqa.selenium.By;
import org.openqa.selenium.Keys;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import java.time.Duration;

public class TestClassB {
    private WebDriver driver;
    private WebElement searchIcon;
    private WebElement searchBar;
    @BeforeClass
    public void setUp() {
        driver=new ChromeDriver();
        driver.get("https://www.w3wiki.net/");
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(3));
        searchIcon= driver.findElement(By.xpath("//*[@id=\"gcse-form\"]/button/i"));
        searchIcon.click();
    }

    @Test(groups = {"nav-search"})
    public void testNavBarSearch() {
        searchBar=driver.findElement(By.xpath("//*[@id=\"gcse-search-input\"]"));
        searchBar.sendKeys("TestNG and Selenium");
        searchBar.sendKeys(Keys.ENTER);
    }

    @AfterClass
    public void tearDown() {
        driver=null;
    }
}

Please Verify your directory structure before proceeding:

Now create the testng.xml file in the root directory of your project (refer the screenshot above). We have already learnt about groups in the previous section. This time we will see how we can include/exclude groups in real project.

testng.xml

XML

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Regression Test Suite">
    <groups>
        <run>
            <include name="page-search" />
            <exclude name="nav-search"/>
        </run>
    </groups>
    <test name="App Test">
        <classes>
            <class name="com.w3wiki.browsertests.TestClassA"></class>
            <class name="com.w3wiki.browsertests.TestClassB" ></class>
        </classes>
    </test>
</suite>

The above document specifies that we want to include all methods belonging to the group ‘page-search’ during execution and exclude those that belong to the group ‘nav-search’.

D. Best practices for organizing tests into logical groups

The three main ways of organizing tests into logical groups are:

Functionality: This means separation on the basis of logical functions. For example, we can have different groups for login-functionality, search-functionality or comments-functionality.
Type: Separation on the basis of type of test is also a good choice like UI Tests, Regression Tests or Smoke Test groups.
Priority: Certain methods may have a higher priority than others. We can assign these methods to groups and run them collectively. Higher priority groups are run earlier than the ones with a lower priority.

Exclude Test Methods

Introduction to excluding test methods

Excluding test methods prevents them from running during test execution. We have already seen how we can prevent groups of tests from running in the previous section. In this section we will look at how we can exclude one test at a time.

Explanation of @Test(enabled = false) annotation

We discussed earlier that the @Test annotation has several properties. One of those properties that we can configure is the ‘enabled’ property. The enabled property is just like a switch on the method. It turns the method on or off as we need. It accepts two valid boolean properties- true and false. By default the enabled property is always set to true.

Example:

@Test(enabled=true/false) public void LoginTest() { //body of the test-method }

A Note on the @Ignore Annotation

The @Ignore annotation is a class level annotation that yield the same output as the @Test(enabled=true/false) feature. The method above could have been stopped from execution using @Ignore annotation as: @Ignore public void LoginTest() { //body of the test-method } This works exactly the same as the code above.

Demonstration of excluding specific test methods using enabled attribute.

To exclude specific test methods using enabled attribute you need to modify the default value of the enabled attribute as:

@Test(enabled=false) public void LoginTest() { //body of the test-method }

This excludes the method LoginTest from execution.

Example code snippet illustrating exclusion of tests

Let’s consider the example we took earlier. We excluded nav-search group from running altogether from the testng.xml file. Now suppose you have many other test methods inside the nav-search group but you want only this method (testNavBarSearch) to not run. Maybe because its incomplete or error-prone or unstable. So instead of excluding the entire group from execution, you can turn this method off and run the group the same way as before.

Example:

Java

public class TestClassB {
    private WebDriver driver;
    private WebElement searchIcon;
    private WebElement searchBar;
    @BeforeClass
    public void setUp() {
        driver=new ChromeDriver();
        driver.get("https://www.w3wiki.net/");
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(3));
        searchIcon= driver.findElement(By.xpath("//*[@id=\"gcse-form\"]/button/i"));
        searchIcon.click();
    }

    @Test(groups = {"nav-search"}, enabled=false)
    public void testNavBarSearch() {
        searchBar=driver.findElement(By.xpath("//*[@id=\"gcse-search-input\"]"));
        searchBar.sendKeys("TestNG and Selenium");
        searchBar.sendKeys(Keys.ENTER);
    }

    @AfterClass
    public void tearDown() {
        driver=null;
    }
}

You can see that in the above example we didn’t even have to remove it from the group. We simply turned it off.

You will see that the output opens only one instance of browser and run gives the following output:

Considerations for when to exclude tests and alternatives

You can exclude tests in the following situations:

Incomplete test
Unstable test
Slow Running Test
Debugging

Alternatively, you can also avoid using this feature by:

Adopting Good code practices
Completing incomplete tests
Fixing unstable ones
See if you can somehow make the slow tests run faster by coming up with a suitable solution

Advanced Techniques for Exclude and Include

Parameterizing test methods inclusion/exclusion

TestNG allows you to pass data to functions at runtime. To know more about test parameterization please visit this page.

Utilizing regular expressions for inclusion/exclusion

To learn more about how you can include and exclude tests using regular expressions, please visit this page.

Dynamic inclusion/exclusion based on runtime conditions

This means that you want to include or exclude certain conditions during runtime based upon some conditions at runtime. For dynamic inclusion and exclusion based on runtime conditions, you could consider using Behavior Driven Development(BDD) style testing frameworks like Spock. This is an advanced topic. This type of technique is mainly used in Robustness testing. You can read more about about Robustness testing here.

Real-world scenarios and use cases for advanced inclusion/exclusion techniques

Debugging: During test development, you could be running into unwanted issues. To pinpoint the exact cause of the problem, you could try running one test suite at a time, followed by one test class and then method. In such a case, you would not want to delete or comment your code at all, which isn’t recommended. So, disabling and enabling tests as per need becomes a crucial feature. This can be easily achieved by setting @Test(enabled=false).
Avoiding Incomplete Tests: There may test methods that you haven’t provided the body for, just left them by defining their method signatures. It would not be wise to run these description less test cases during tests. It would be a better option to disable these tests and enable them after you are satisfied with them. The syntax to use would be @Test(enabled=false).
Skipping poorly written Tests: When deciding to run entire test suites, it is a good option to disable tests that aren’t well written and produce ambiguous results that could make the execution of entire test suite fail. These include those test cases that don’t always produce the outcome we expect. Use @Ignore annotation or @Test(enabled=false) to skip the test.
Testing Specific Features: Suppose you have to perform tests on a particular feature or functionality. And because your application is big, you have made these tests in separate groups. In such cases, you can easily disable other tests groups and test your desired feature. The syntax to add a test in a group is @Test(groups={“login”, “cart”}). To exclude the group of feature cart, you simply need to use the <exclude name=”cart” /> tag in testng.xml file. This would allow tests in the login group to execute and the ones in the cart group to be ignored.

Conclusion

In this article we covered a lot of things. We looked at how we can configure the testng.xml file to run only the tests that we want. We also look at how we can use the @Ignore annotation and the @Test(enabled=false) to produce the same results. We looked at how we can create groups and include or exclude those specific groups inside the testng.xml file. Hope you have learnt something useful from this article. If you have any queries, please feel free to discuss in the comments below.

FAQ’s on Exclude Include Test Cases in TestNG

Where can I use the @Ignore annotation?

Ans: The @Ignore annotation is both a class and a method level annotation. If you wish to ignore an entire class, use it at the class level. If you just want to ignore a method, annotate the method with this.

Where can I use the <groups>…</groups> tag inside testng.xml?

Ans: You can you the groups tag either inside the <suite><suite> tag or the <test></test> tag only.

Can I rename the testng.xml file to anything else?

Ans: No. The testng.xml file is a unique one and if you change it to anything else other than the regular expression allowed, then it will not be detected by the test environment and you will not be able to use the file to run your test.

Can I place the testng.xml file anywhere I want?

Ans: No. You should not place the testng.xml file anywhere you want. It should only be placed at the same level as the pom.xml file in order to be detected by the test environment. Also make sure that the test classes follow the tree structure as depicted in the article.

Tags:

#TestNG #Automation Testing #Testing Tools

Test Driven Development using JUnit5 and Mockito

Basic Commands in Cypress