Automation isn’t just regression, Look Around

Scene 1

One fine Monday morning, I walked across Mark who is a testers and a domain expert in the application to which I am pretty new.

ME: “Hey, café?”

Mark: “No dude just received this list of loans which I need to feed into the test system ”

ME: “Oh is it? What you do actually?”

Mark: “Here is the list of 100 odd loans that I have to feed into this using this XYZ application, so that ABC department will test this out in the build they get tomorrow.”

(Mark is the domain expert in this XYZ application, many other application talk to his application to get data for reporting, rule engines to name a few)

ME: “Hey that should be boring, is a onetime work?”

Mark: “No ME, I’ve to do this whenever any other department need some specific loan data. I have to do this almost every week for one or the other department during the release cycle.”

ME: “oops! So what exactly other departments give to you?”

Mark shows an excel sheet that has just list of loan numbers in a single column

Mark: “They just email me the list of loans they need, I just use the loan number to this main DB machine and gives the source DB so that the loan from this gets copied to the test DB.”

ME: “Is it? Can you give me a demo for it?”

On seeing the demo I understood that he navigates 4 mainframe screens then use the loan number provided and the test DB server provided by the other department to initialize jobs. He repeats this for the entire list of loans, each takes 3 minutes to enter.

ME: “What if the loan is already present in the source DB?”

Mark: “You get a message about the existence”

The first think that flashed is better to automate this. We already had the most costly automation tool with all add in for the mainframe interface. Wrote a very basic script to read the loan number and designation DB name from QTPs Data table and automate the other steps. Now each loan takes less than 30 seconds, asked him to maintain a sheet which calculates how many hours was by this script. The effort to create this script was half a day. We faced few issues in new release that were fixed using regular expression.

Next Monday morning

ME thru messenger: “Mark, café?”

Mark: “I am working from home, have your café alone”

ME:  😦

Scene 2:

Another not so fine someday morning, meeting with Mr.X.

Mr.X: “We need to come up with automation scripts for the entire regression test suite and deliver to the client”

ME: “Automation suite to client? But why do they need?”

Mr.X: “We have made agreement with client that we provide with complete regression automation suite so that they can run complete regression suite through automation”

ME: “But I think from the client perspective they need to test manually so they get the feel of the application, also from our experience we know very well that the automation scripts need to be debugged very often by experienced automation engineer for many problems starting from Data integrity issues to understanding the results to many such problems”

As usual without listening, my Mr.X continued

Mr.X: “We need to build a robust framework and so that the client will be able to use our framework easily”

ME: “So are we hiring another QA team to test this project, this looks like you are building software to test this software”

Mr.X: “Not exactly, but we need to look into next step in automation”

ME: “That is perfectly fine but at what cost? Whose benefit? ”

I am seeing this trend happening in many places. I really don’t understand the purpose of building a GUI automation regression scripts to be run by the client. First the trend was Regression Automation now it has another version GUI Regression Automation Suite for Clients.

But the problems here are

  • As Client I would love to see\test the system that was build, so that I can make a judgment whether the system is behaving as expected. I do not how automation adds value to client.
  • And no one is considering the maintenance cost. Automation even on play back needs experienced person to monitor and debug it regularly
  • And the time automation tool takes to yield the returns. In Most cases it will take three or four years to yield the returns. Refer this link to understand why

I gave those two of those incidents to emphasize that

  1. Automation isn’t just automated regression testing. Automation and regression involve different considerations as said by Dr.Cem Kaner.
  2. Look around and talk with other people to know the kind of work they do, so you may add some value to it. Also you might get some interesting ideas from them to implement in your work.

Happy Looking Around  🙂



9 thoughts on “Automation isn’t just regression, Look Around

  1. >>> I do not how automation adds value to client.

    Be careful not to reify “value” (a multi dimensional attribute). Through dialog and questioning, you can discover client’s value system. You can, in most of the times. Whether that is a “good” one or not… that is a context specific question. Depends on who you are and what you are trying to do

    In this case, the Mgr is simply trying to fulfill an obligation that someone else has made (mostly verbaly) about providing automation regression suite so that the end users can test the software themselves without investing in extra resources. I believe this is a kind of customizable software … say SAP…

    So, providing a regression automation suite with the application becomes a selling proposition.

    1. Shrini,thanks for reading through the post and sharing your view about the value as a multi dimensional attribute.I completely agree with that.
      Coming to, providing a regression automation suite as a selling proposition,in most cases due
      a. obligation was made without considering the effort and time that needs to complete such tasks.
      b. when that comes down to managers who are not from testing background.
      c. when clients and mangers are made to think that automation is a piece of cake by tool vendors Salespersons(who generally show the classical login examples 🙂 )
      the automation tasks become a sideline activity.At the end of the project the result achieved is nothing or we are not able to showcase the value we have added through such automation tasks.
      I had even involved in a project where I worked just two weeks forced to create some scripts and deliver to the client as it was committed.

      But I was able to show case value added to the client (the time saved) that I made to my client by automating the test data creation.

  2. Great Post – completely agree on adding value and conversating with others – help others grow and make others feel great.

    about giving regression suite to the client – I would certainly feel hesitant too. You asked the right question – to hire another team to test the automation tool. Depending on the application under test (AUT), certainly the automation needs a lot of data preparation to feed with.

    another concern is, sharing the source code, if it is ok to share the source code for the test suite. if it is, then it would have to into project stream, budget, quality, risks, impacts, and all other parameters would have to be considered and can’t be blindly shared with client. What if a change request to AUT happens, who would maintain the automation suite for those changes – this means the automation tool would fail for those test scenarios where a change request or a bug fix had impacted.


    1. Thanks Ram for sharing your views.
      What ever questions why raised against the maintenance data preparation,budget,quality were never considered.,and because of that automation becomes a sideline task and at the end nothing solid was achieved.

  3. Currently I’m working on a project that is using an automation tool to do validation/verification of a system that has been upgraded. I’m not totally testing the system, only smoke tests as part of it all, but my scripts go into the system and grab configuration information before and after the upgrade to make sure the process went correctly and that things didn’t get mangled during the upgrade process.

    This is being used by the client with their customers to help improve and speed up the upgrade process. Time is money.

    1. I would like to know of is that a GUI automation? I would be great if you can elaborate what kind of tests you do with the scripts?

      1. It is both GUI based and backend (server & database). The scripts are run via an independent VB.Net applet I built (I use the test tools API to call it outside of the Test Mgmt. tool). Based on user input the different tests are run to either perform a light GUI smoke test (make sure different apps. and functionality is available) or a configuration type test (either GUI or backend) to make sure the upgraded system is still much the same as before.

        This before/after imaging of the system is designed to help the people doing the upgrade to quickly determine if there are problems due to the upgrade from a data and configuration standpoint. Doing it all by hand (as is currently being done) is very time consuming and error prone. This tool is being built to increase efficiency and reduce rework.

        Not your typical use of a GUI Automation tool eh.

        1. That’s what I want to know,the problem is with the GUI automation tool.Many just go blindly behind the GUI automation tool without understanding their need and also what the tool is capable of,and some times the expectation out of this tool is very high,with out considering the technical expertise of testers.(will be posting about that soon)
          Thanks Jim for sharing your approach.

  4. “the time automation tool takes to yield the returns. In Most cases it will take three or four years to yield the returns.’ – I find hard to digest, as many of current generation application either goes under make over or get a lot of new features and functionality.

    Test Automation helps greatly for repetative tasks, but they themselves have a very short lifespan with high pace changing dynamics. Over 80-85% of automation tests becauses useless with 2 years, as per my experience.

    Be it Manual Testing or Automated Testing, one has to ensure that nothing remains constant and hence would need updates, pitching on long term gains from scripts could be bad decisions

    Over all interesting article. 🙂
    Best Regards
    Alex Jones

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s