rajasankar

In this post, answers for the questions such as what is data driven testing?, how that is important to test automation and how to get that data need to test? are discussed.

What is data driven testing?

In manual testing, using multiple values for the same course of action. In automated testing, using multiple values for same test script.

How this is important to test automation?

Using this method, testers can create highly maintainable test scripts. Additional data can be added to the data source(excel,database) with out modifying the test script.

How to get the needed data?

Using the black-box testing techniques such as Boundary value analysis, Equivalent partitioning and Error Guessing. You may read about these. However, lets see that.

Equivalent partitioning

To test the software with inputs from correct and incorrect data. For example, if you're testing a filed that will accept only positive integers. Positive integer, negative integer and alphabet inputs consists of three portions of data we need to test. With the equivalent partitioning method, one data from each portion is enough.

Boundary value analysis

In this, data in the boundary of the partition is take. Testing a field which accepts data from 10 to 20 will be tested with 9, 10, 19, 20, 21.

Error Guessing

This method relies on testers knowledge about the product,intuition and past experience.

Now, using these methods, the data is generated. However, we need to answer another question before starting this. Is the necessary to test the product with all these data? Is the possible to reduce the number of combinations needed?

Pairwise testing the answer for that question. Before trying that out, you can take look at Cem Kaner's Impossibility of complete testing. With that background work, you can prepre the data for the minimum set of possibilities needed test. Take a look at this website more info on pairwise testing. You may interested to check James Bach's All Pairs test case generation tool too.

Benefits of version control in a software project doesn't need any explanation. Adopting the same in test automation will benefit even teams have more than one member. In this post, we can see, what are all the files needed to be checked in the version control and how to maintain the setup.

Most of the files generated by QEngine can be opened using a text editor. Only files with QED extensions are binary files. You can only import those files in QEngine.

When you create a suitea folder with the same name created under /projects. It contains conf,webscript folders.

Conf folder - Has all the configuration related files
dataset - Contains data files
webscripts - Contains script files. Every script created will have a folder with the same name, which contains .wcs file and a map file if the script uses local level map. Information about the recorded objects is stored in an xml file which will be located in the conf folder, if the script uses global level map.

All the three folders can be checked in the version control system. If you're using the custom scripts files which will be located in /jars folder, you can check in that too.

You can replace the files in the same directory, if the original ones are lost or if you want to revert to the previous entires.

Rajasankar
rajasankar at zohocorp dot com

NetBeans released support Python recently. Check the features here http://www.netbeans.org/features/python/index.html

I've tried to use that IDE for editing QEngine custom scripts. Here are the steps,

1. Download and install the EA from here, http://dlc.sun.com.edgesuite.net/netbeans/6.7/python/ea2/
2. Create a new project. While creating the project, include the /QEngine home/jars folder in the Python path
3. Wait for NetBeans to scan and index the added folder
4. Now, you can use the IDE editor for editing custom script files including code completion.

Try and let me know.

Rajasankar
rajasankar at zohocorp dot com

Maintaining scripts is an important task in UI automation. Some times, it may be difficult to do. However, if we separate script flow and actual actions, we may able to maintain the scripts without any issues even the UI changes.

The formal approach.

Here is the steps to create for a test script w.r.t to QEngine

1. Create New Suite named as Test
2. Create New module in that suite if needed
3. Create New Script named as TestScript
4. Record the steps in TestScript.
5. Playback when needed.

Looks cool. Say, we have 100 scripts in that Test Suite. Playback is not an issue. Assume that one element changed in the UI. Now, we need to check and edit all the 100 scripts. This will neutralize any gains made using test automation.

Here is the new model

1. Create New Suite named as Test
2. Create python file for UI functions named UIFunctions.py
3. Create python file for script flow named ScriptActions.py
4. Create New Script named as TestScript
5. Record or use export mode for recording the scripts. Create functions using recorded actions. Store the functions in UIFunctions.py
6. Call the functions needed in ScriptActions.py.
7. Call and import the script flow in TestScript.
8. Playback.

In this approach, if anything changes we need to modify in two files. Let see this with examples.

Content of UIFunctions.py

Code:

from Framework import *

def clickthisbutton():
setLastWindow()

def clickthislink():
setLastWindow()

def function1():

..........
.........
.....

Content of ScriptActions.py

Code:

from UIFunctions import *

def Script01():
clickthisbuttion()
clickthislink()

def Script02():
clickthislink()
clickthisbutton()

def Script03():

def Script04():

..........
.......
.......

Content of TestScript01

Code:

from ScriptActions import Script01

Script01()

Content of TestScript02

from ScriptActions import Script02

Script02()
.......
.....

In this approach, there is no need to edit the script files,even though they can be editied in the text editor. The files has .wcs extention and located in /QEngine Home/projects//webscripts/ folder.

If the UI changes, you need to either edit UIFunctions.py or ScriptActions.py or both. If needed you can use an macro to replace the changed parameter.

This approach combines the advantage of Scripting with Playback. If you wish, you can use the texteditor or IDE to edit the python files.

Rajasankar
rajasankar at zohocorp dot com

Once, we embrace the idea of removing the object repositories from the automation, we can solve few issues to.

Say, we need to click on the "Add This", what are the options we have?

Code:

clickElement("Add This",1)

Here the property of "Add This" stored in object repository.

Code:

fireEventOnElement("ID","addthis","Add This",1,"Click","NONE","false")

Here the property stored in the python file.

Code:

fireEventOnElement(".*","innertext","Add This",1,"Click","NONE","true")

Here the property stored in the python and if anything changes in the tag or property of the object except innertext, this function will be played. Why only innertext? Innertext will be the one exposed in web page which means if the function didn't play back, we can identify that easily.

What about the index or more than one Add This is present in the page?
If we know that index of those Add This, we can write that function based on that. If that more Add This may added in future, we can use

Code:

getElementProperty(tagName,propertyName,propertyValue,index,propertyNeeded,regExpRequired='false')

A simple loop like one below can check that.

Code:

i=1
while i:
value=getElementProperty(".*","innertext",".*",i,"innertext","true")
if value=="Add This":
maxvalindex=i
i=i 1
else:
break

Using getElementProperty you can get the if any Add This property has been added and plan accordingly.

Rajasankar
rajasankar at zohocorp dot com

Every UI automation software uses some form of object repositories. Before diving into the need of those, first let us know what is an object repositories and how it affects UI automation.

When you record a action in the UI, the information about the object such as type,action performed,index in the page,input made if any and window name will be stored in a file. That file may be xml, excel,database,custom extension or binary format. At playback the object info is read and software will try to find the same object in the page to perform the same operation. The file in which the object stored is called Object Repository.

This object repositories work best when the UI doesn't change or web page has only static content. When the UI changes or the properties of object changed w.r.t page or entirely, then playback wont perform the action on that object.

Even if we assume that object properties is not changed, the reading and executing the test will take more time and will grow exponentially with number of objects stored. Whatever the file format, automation software need to read the file, search for the object and execute the desired action.

As most of pages has dynamic content and number of pages for any web hosted software increasing how do we tackle this issue?

My solution is simple. Get rid of Object repository or reduce the number of objects stored in that to minimum(subjective). Is that possible? May be.

QEngine has few functions to do this.

Code:

fireEventOnChildElement(tagName,propertyName,propertyValue,
index,identifier,actionName,actionValue='',childpropname='',
childpropval='',childindex=1,childRegExp='false',parentRegExp='false')
fireEventOnElement(tagName,propertyName,propertyValue,index,actionName,actionValue='',regExpRequired='false')
fireEventAtPosition(title,classID,xCoord,yCoord,actionName,actionValue='',waittime=1)
fireEventOnObject(classID,type,xCoord,yCoord,actionName,actionValue='')

In these functions, property of the object has been stored in that python/WCS file. So, there is no need to read from the another file when playback. That will improve the performance into great extent.

Hey, wait a minute, you didn't answer what if the object property has changed?. Well that will be answered in second part of this post.

Rajasankar
rajasankar at zohocorp dot com

ScreenCapture in UI Automation

In UI automation, how to know if the script played correctly?

One way is check the log messages and the check the output. That will be an cumbersome process to do. The better way is to see screenshots of the page. QEngine has a function to do this.

Code:

saveScreenShot(outFileName,waittime)

You can call this way using the python function, if you need to generate filename automatically,

Code:

import time

def screenshot():
filename=time.strftime("%Y_%m_%d_%H_%M_%S")
saveScreenShot(filename,"1")

There is an one limitation. This saves the file as a JPEG format and save the files in dataset folder in the projects.

You can write an Java class to save the images in the PNG format which will occupy less space than JPEG file.

Here is the code

Code:

import java.awt.Dimension;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.Toolkit;
import java.awt.image.BufferedImage;
import javax.imageio.ImageIO;
import java.io.File;
import java.util.Calendar;

public class SaveScreen {

public static void main (String args[])
{
}

public void saveScreen() throws Exception
{

Calendar now = Calendar.getInstance();
int year = now.get(Calendar.YEAR);
int month = now.get(Calendar.MONTH);
int day = now.get(Calendar.DAY_OF_MONTH);
int hour = now.get(Calendar.HOUR_OF_DAY);
int min = now.get(Calendar.MINUTE);
int sec = now.get(Calendar.SECOND);
String fileNameone = "Screenshot_" year "_" month "_" day "_" hour "_" min "_" sec ".png";

Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
Rectangle screenRectangle = new Rectangle(screenSize);
Robot robot = new Robot();
BufferedImage image = robot.createScreenCapture(screenRectangle);
ImageIO.write(image, "png", new File(fileNameone));
}

}

You can also change the

Code:

String fileNameone = "Screenshot_" year "_" month "_" day "_" hour "_" min "_" sec ".png";

into

Code:

String fileNameone = foldername "Screenshot_" year "_" month "_" day "_" hour "_" min "_" sec ".png";

if you need to save the files in the required folder.

Refer to the previous blog entry for how to import your own code in the QEngine.

Rajasankar
rajasankar at zohocorp dot com

As you know QEngine based on Jython(Java implementation of Python), you can write Java code and import that in QEngine.

Code:

public class test
{
public static void main(String args[])
{
}
public void work()
{

}
}

you can import that like this

Code:

import test as te

t=te();

use the method mentioned there

def makework():
t.work()

If you need to use this method across multiple scripts, you need to create and python file under /QEngine/jars folder and import that in the scripts file

Code:

workfile.py contains

import test as te
t=te();

def makework():
t.work()

Use

Code:

from workfile import *

Code:

from workfile import makework

Finally add the class in the classpath. Add this class entry to //bin/StartWebTestServer.bat in "set CLASS_PATH=" line. You need to restart the server after this change.

Rajasankar
rajasankar at zohocorp dot com

If you encounter the dynamic windows in the UI automation, you can easily handle them.

Here are the functions available

Code:

setDynamicWindow(parenttitle,parentindex=1,framepropertyname="NONE",framepropertyvalue="NONE",frameindex=1)
closeDynamicWindow(parenttitle,parentindex=1)

If more than one window is present you can use parentindex and frameindex to choose which window.

However, consider this situation. You don't want to enter the title of the window or title keeps changing. What to do in this case?

We need to use the following functions to handle such cases.

Code:

getLastWindowTitle()
getWindowTitle()
getWindowURL()
getLastWindowURL()

I hope the function names are self explanatory. However, we need to use setLastWindow function once we changed the focus.

Example 1.

If the dynamic window doesn't have any inner frames.

Code:

setDynamicWindow("Test page",1,"NONE","NONE",1)

closeDynamicWindow("Test page",1)

Example 2.

If the dynamic window has any inner frames.

Code:

setDynamicWindow("Test page",1,"title","mypage",1)

closeDynamicWindow("Test page",1)

Now consider this case. Windows title is not known or can't be predicted.

Example 3.

Code:

gwt=getWindowTitle()

gwtn=getWindowTitle()
if gwt!=gwtn:
setDynamicWindow(gwtn,1,"NONE","NONE",1)

closeDynamicWindow(gwtn,1)

Code:

gwt=getLastWindowTitle()
gwtn=getWindowTitle()
if gwt!=gwtn:
setDynamicWindow(gwtn,1,"NONE","NONE",1)

closeDynamicWindow(gwtn,1)

You can add more conditions if needed

Example 4.

Code:

gwt=getLastWindowTitle()
gwtn=getWindowTitle()
if gwt!=gwtn and gwt!= and gwtn!=:
setDynamicWindow(gwtn,1,"NONE","NONE",1)

closeDynamicWindow(gwtn,1)

This will ensure that dynamic windows are taken care without knowing the window title.

Rajasankar
rajasankar at zohocorp dot com

There is an endless debate going on in the testing world regarding the strategy for automation testing. What to use, scripting or record/paly back tool?

Before commenting which one is best, we need to understand what are the pros and cons these two.

Let us see pros and cons of scripting

pros
1. Standard scripting language and standard library
2. No vendor scripting
3. Version Control
4. Running with own testsuite or with standard library

cons
1. Need to build from scratch
2. Time consuming process
3. Suited for individual need. May not be applicable to other software
4. Need to identify the elements from source manually.

For record/playback tool,

pros
1. Easy identification of elements
2. Maintenance will be taken care by vendor
3. Will come with utilities to run the test suite
4. Easy setup and running

cons
1. Maintaining the scripts is difficult.
2. Version control may not be available.
3. Vendor scripting may have a deep learning curve

Both the options offer some advantageous. If we can combine both, we can reduce time and do the things in a hassle free manner. That will offer best of both worlds, then it is easier to do automation, Isn't?

So the ideal tool should have the version control, minimal vendor scripting, options for one or more standard scripting languages. Next time when you evaluate an tool, check these options in it.

Rajasankar
rajasankar at zohocorp dot com

Data Driven Testing

Using Version Control in QEngine

Using NetBeans IDE for QEngine

Separate Scripts and Functions-Script Maintenance in QEngine

Do we still need Object Repositories in UI Automation?-II

Do we still need Object Repositories in UI Automation?-I

ScreenCapture in UI Automation

How to write your own Java code in QEngine.

Automation Tips using QEngine-I. Using Dynamic window.

Which is best for Automation? Record/PlayBack or Scripting