iTest

Last week I participated in a competition conducted by 99tests.com, a crowd sourced testing start-up from India.  This is  my approach\experience report on how I tested and won the competition.

What were the givens?

No requirement, no design documents, no test cases, no use cases, no user stories. All I had were credentials to login and the URL to test. So, do I need to agitate and not to start testing until I get requirements? Do I sit and write test cases? Any conventional scripted testers would’ve struck wondering what to do. So, here is a warning for all those scripted testers it looks things are changing because of agile approach and such crowd sourced testing services.

How did iStart?

I didn’t waste any time exploring the application. I thought the best approach with the givens (actually no givens ;-)) would be, jump straight into using the application and observing the behavior. I thought that would help with forming test ideas and then to build from there.

Being a user of such online shopping sites helped me to frame some initial expectation. So, started with Follow the (user) Flow heuristics. I decided to create my own credentials instead of using test credentials and login. Then observe how easy or buggy it is to find an item, add to cart, check out, paying through a third party payment gateway and choose a shipping address of my choice.

Did iFollow the flow?

Registration was successful  upon giving correct details. But I didn’t get to see any error message, so decided to enter some invalid data to find the application behavior. What I found in giving invalid input was the error messages were not user-friendly, not just in terms of the message, in terms of usability as well. Error messages were in a different page and users needing to click back button to get back to the registration page. On click on back button the entire data entered were lost. So, this made me take a deviation from my initial plans of Follow the flow. Decided to test this module thoroughly, wondering why? Read bug advocacy of this bug below.

How iDid bug advocacy?

I didn’t just log the bug with a summary line, description, steps to reproduce and screen shots. Instead, I also explained how and why this might bug the users and impact of the bug.

The user registration page plays a vital role in giving the first impression about the application behavior. Also, users use online shopping sites mainly to save time, by displaying errors in another page and asking the user to come back to previous screen to correct the data is actually wasting sufficient time. This advocacy helps in understanding the real impact of such bugs. So, always explain how any bugs\issues found would potentially affect the user experience.

Your bugs are your representatives. The bug logging also depends on context, if you know your developers very well and if he is sitting next to you, the bug logging, or advocacy may differ from the way you log bugs in crowd sourced testing. Here you have no clue about, who is doing bug triage, developers and their understanding of the product. So always give as much detail as possible so that they can’t reject your bugs.

Always remember this  most famous movie dialogs of all time, from “The God Father” movie while logging the bugs

“I’m going to make him an offer (details) he can’t refuse (to fix).”

Did iJust log bugs?

No, I went through most of the bugs logged by other testers. I posted comments, raised questions where ever I felt the bug was really not a bug or if the priority was inflated or if the issues were duplicated. Also, I neither missed to appreciate some good bugs reported by other testers nor missed to learn from fellow testers. I was actually a little disappointed that there were a significant amount of bugs without clear description and duplicates.

What were my objectives?

Than winning the competition, my objective was to log the maximum number of valid bugs. Ended logging 50 valid bugs, maximum by any tester. Also wanted to maintain a high bug acceptance ratio, 86% of the bugs were valid. Happy with that but still wanted to improve on acceptance ratio.

So, What iDidn’t test?

I found few even tested the Facebook ‘Like’ gadgets, Payment gateway and spell checks in the application’s blog. Though few of such bugs were valid, those were not from the application under test, so they got rejected. So,

Know your boundaries, so you do good enough testing in given time.

Ready to try some bug advocacy?

Here are a few other bugs logged by me, try to advocate for them. The application under test was an online shopping site.

  • “Similar Items” feature is missing.
  • Same book title, but displayed with huge difference in price tag.
  • The amount should always be right justified.
  • On entering the special characters the system through an “invalid gift message”
  • Pre order items are shown as Available and Buy Now.
  • Is the final price displayed in product description inclusive of taxes?
  • Can’t store search results.

Happy Bug Hunting!

P.S.

Some of my favorite lessons in Bug Advocacy chapter of “Lessons Learned in Software Testing”  book

55: You are what you write.

58: Your bug report is your representative.

83: The summary line is the most important line in the bug report.

89: Use market or support data when appropriate.

101: When you decide to fight, decide to win!

Test[ing this] Case

The first thing flashes in most software testers mind immediately on hearing software testing is Test Case. Even most of us explain our work to others and even to fellow testers something like this “I understand requirements, write test cases and test the application, log the bugs and mark the test case pass or fail based on the expected results”. I too started my testing career by explaining what a test case is and SDLC model that was (never) followed to an interviewer. But soon, I realized that it is practically not possible (or I am not intelligent enough) to write most test ideas (cases) even with detailed requirements. Most test ideas don’t popup while analyzing the requirements, but while performing the real testing on the product. The test cases written with requirement documents are only based initial abstract understandings made on theory, but the real practical ideas come while testing. But even recently many people advice to write detailed test cases, that even a layperson can understand. Are they recruiting a layperson or a smart tester? Why still the main focus is with writing detailed test cases even in agile world? Looking back the water fall model, what I can think of reason for writing detailed test cases is, there was  a phase in SDLC were developers are busy writing and developing a product for months together, before delivering into testing team. So, probably some smart manager got this idea of test case writing to keep the testers engaged during that phase. Few rituals still followed without analyzing, if those suit the context. 1.  Write detailed test cases for the projects where there is good clarity on requirements. We got clear requirements so write detailed test cases. 2.  Write detailed test cases where there are no clear requirements . The requirements are not clear let use have detailed test case document. 3.  Even if the project has very tight deadline, a certain percentage of time is allocated for test case writing instead of focusing on real testing . Because Test case documents are (mis)believed to be the only Savior of testers. 4.  I’ve even seen reverse engineering of coming up with test cases from a well functioning application, with the help of help files 🙂 5.  There is something called traceability matrix, which traces all the detailed test cases back to your requirement, I think this is designed especially to say the whole world, see we’ve spent so much of man power to duplicate the requirements and here is the proof for that 6. And test cases are again considered as Savior in this job hopping era, but reality is forcing to write such detailed test cases and asking the testers to blindly execute just the documented test cases too are the also one of the reasons for job hopping. Why a similar kind of test cases is demanded everywhere for every context?Why do test case document looks similar where ever I go? 😦 . The only standard that is followed more or less correctly in testing world is test case template. The problem with such detailed test cases writing and following them is, soon the application will develop resistance against bug, because testers were focused only on what is written in the test case nothing less nothing more. Any tests executed blindly by a human just against expected result is as good as (as bad as) automated test checks. Such practices never help to spot the Black Swan* [The disproportionate role of high-impact, hard to predict, and rare events that are beyond the realm of normal expectations] you have to explore more with real application to spot the Black Swan. Testing according to James Bach Testing is the infinite process of comparing the invisible to the ambiguous so as to avoid the unthinkable happening to the anonymous. So, let me try to form a different Black Swam theory, as a tester our job is to predict and spot the Black Swan before the customers or client spot it. 🙂 Even writing test cases need exploration, but it becomes a concern when the explorations stop (or forced to stop) once test case are written. So, constantly come up with new ideas to add more value. Instead of wasting time in writing detailed test cases cheat sheets like this  Test Heuristics Cheat Sheet would cover most of the basic test cases, why to write again? If you feel You Are Not Done Yet, then check this You Are Not Done Yet. And choose SBTM or any relevant format that suits the context. Any available basic requirement document plus the above cheat sheet should be more than enough to do a good enough testing.

Testers aren’t  for documenting, they are for providing valuable information.

P.S: Can you recollect  the color of each letter of Google logo in its home page? (Of course with no doodles), the page most of  us visit every second hour. If you can’t, then you are affected with major Black Swan blindness.

Happy testing your Case!

Dhanasekar S.

* I’ve just started reading The Black Swan book by Nassim Nicholas Taleb. Yet to read it completely.

FaceBook Logoff

I have this habit of logging into any shopping site like Flipkart with same ID, same time in two different browsers (Firefox and Chrome). With one of the sessions, I browse through the items, and other one to add to the shopping cart. This is very helpful to me, to do shopping faster.

I thought I would do the same with Facebook as well. I would use one of the sessions to read all the status updates and other session to chat or render the videos and watch, so I can browse through all the updates from friends in quick time. But I observed logging into second browser forces to log out from the previous browser session

Is this a problem?

I don’t think as a tester you can decide if this is a really a problem assuming either one of those scenarios as an industry standards ;-).Even if one of those scenarios is set as an industry standards it is not necessary to follow them blindly. So Face off 1 yourself as a user to analyze such scenarios.

We cannot think of everything before we start.

You don’t know what you want until you see it.

While shopping user (like me) has two sessions opened. I am done with my shopping .I do check out after adding all the items to the chart and make payment, check my mailing address,which  takes close to 5 to 10 minutes. And then I log out of this session but completely forgot about the other session opened in different browser, and if the user is using a shared computer which is very common in India. So, here it looks it is better to log off all the sessions in the machine if user log off from a session. But one way it is helpful to user if he can access more than one session at a time.

So, which is the correct behavior? It is always decided by what your users are up to. So face off like a user and analyze the observed behavior.

Even let say Flipkart decides to change this and allows only one user session similar to Facebook by maintaining the sessions in server instead of cookies. It should be communicated to user in a very positive manner by highlighting the security risk. Any change like this cannot go without educating the user. Users who are very used to such behavior may not accept those changes.

User satisfaction comes from handling change, not mitigating risk.

Wait. Did I say it is the user always? No(w), I am seeing some new terminology being used these days apart from user friendly, its advertiser friendly ;-). Sometimes you have to consider advertisers as well, so no harm in annoying user for some time.  There you should not think about user friendly close buttons.

Aware of your potential user\customer very well, to analyze what you observed is really a problem. None of the so called industry standards or testing techniques like orthogonal array, boundary value analyses or whatever it is, will help to capture or analyze such scenarios, and we can’t estimate for such analyses in advance. So

Apply your sapience and heuristics. Don’t follow rules.

“Hell, there are no rules here, we’re trying to accomplish something” ~ Thomas Edison

Note to Readers:

  • Here I have given just one possible scenario that I generally do as a user, I like you to bring\think in different perspective to analyses.
  • I had a vague idea about this log off session, was thinking on how to shape this in to an article. Finally this got this shape after  reading this wonderful post by Nathan Smith Estimation is bunk

1 –  Face off (movie) – A revolutionary medical technique allows an undercover agent to take the physical appearance of a major criminal and infiltrate his organization.

How to measure Productivity?

This happened in one fine day in a team meeting. One of my co-workers raised her doubt on measuring the productivity.

A Tool to measure productivity?

She picked up a task and based on her initial understanding estimated that the task would require two hours to complete. When she started with the task she ended up in exploring for solution and a bit of research for about six hours and learned that two lines of code will fix the task. So she coded those two lines complied in ten minutes. Now, what is the productivity and how to measure that?

I really don’t have an answer,as I can’t quantify the productivity to a mere number and I believe most development and testing tasks are like this. Do you have an answer?

Addendum added an hour later:

Every one is forced to show their productivity,that is the only measure any company needs. Will KLOC ,Burn Down Chart or any other metrics show the productivity correctly here?

Addendum added After Aravind’s Comments:

I liked all the points in comments by Aravind  below

Share

Blogger Is A Time Machine

Did you miss to participate in an event?

Not to worry, Blogger will take you to past, to attend the events.

Look at the screenshot, it was taken on Oct 11, 2009.There is an event that is happening on Oct 10, 2009 IST 11.00AM. Here I have an option to select I’m Going and Not going, both the options are enabled. I can click on them 🙂 to attend an event that is happening (happened) in past.

NOTE : Not sure about the time zone kept in the blog,but all the time zones were ahead of the event time, when this screen shot was taken
bloggerbug

This is one of the most undocumented feature of blogger uncovered by me 😛

Another problem faced during signing in.

To participate in the event, it asked me to join the Site to RSVP, what does this RSVP mean????? 😦 . I clicked on the link to Join RSVP (without knowing what that mean, I don’t expect every user should Google this to find, what is RSVP?). A pop up opened and there was another message which says “Sign In or Join us using the link above”

whereisthelink

Sorry Blogger! I can’t see any link in the roof :-(, where is the link????????????

I found only a Follow button; I don’t think all users will click the button when you are say click the link.

Ok, after clicking the link (button) and successful sign in, I was given an option to select to go to the event (I’m going).I clicked on that link and it was successful.

Oh! Wait, I am still given option to join the event in the main window, whatever I did was in a pop up, so how many times do I need join the event??????

twice

Oops! Wait for some more bugs, IE showed me Google Calendar option below the event, which is missing in Chrome, Google against its own product??? Interesting

IE

googlecalendarIEChrome

googlecalendarchrome

Like to thank Enjoytesting for this, I was doing all those in his blog, I doubt whether he enjoyed this 🙂

Addendum added while drafting this 🙂

WordPress.com’s turn 🙂

Found this while inserting screenshot into this post

I have -6 pictures in my gallery,some body help me what does this mean??

negative

Happened when I was deleting pictures from my gallery,I keep clicking the delete option this happens.I think the sytem is counting the clicks I make and subtracting from the pictures present in the gallery.Smart uh!

<Decided to End this post>

Click here to see more interesting bugs under this category

Monster Buggy

Found one interesting or an unpleasant bug in monsterindia.com jobportal. It made me to check back my basic maths skills 🙂

Just look at the screenshots below

monsterbug1

Required Experience : 3 -10 years

Applicant Experience : 5 yrs and 3 months.

As per my knowledge I think I am eligible to apply for this,but look what the system says “Your following information did not match the job requirement ” 😦

I was little doubtful because this might be some odd case,but it happens consistently,see another screenshot.

monsterbug2

I am really shocked to see this ,was the system tested throughly? Because this is a simple positive condition that should have been tested.

Where is the problem?

Is it in test data ? I dont think so because user can enter his experience only be selecting the years from drop down,so there is no data format error

But still how did they miss this?

Did you experience the same problem?

Click here to see more interesting bugs under this category

Power Of Exploring In Software Testing

I am not here to explain in detail about exploratory testing . Please read James Bach, Cem Kaner and Michael Bolton to learn about exploratory testing.

This is just to share my experience how exploratory approach is powerful in finding bugs and solutions in software testing. This is about my experience in getting a solution to a problem by exploring. The same holds true in finding bugs as well.

Let me start off with Bangalore Weekend Testers (BWT), to know more about BWT check here and hereBWT 9 was my first experience of testing with a group of enthusiastic testers. We tested an online photo editing tool splashup.I would say this really tested my inexperience towards reacting to some unprepared situations. Though I didn’t come out with flying colors, but was able to react quickly and found some interesting bugs (at least for me). Here is my test report. This really helped me to analyze and understand my testing skills. So my testing skills got tested 🙂 in BWT. I think every tester has to experience this and evaluate oneself. I had done a lot of buddy testing in my workplace, but still this is the real buddy testing, because what I do in my organizations is a restricted buddy testing 🙂 .

Coming back to exploratory testing (ET), I am always interested in doing ET as it is always fun, interesting and a real learning experience to become a true tester. After finishing my BWT 9, I read a post by fellow tester in BWT Amit Kulkarni, about why the error messages does not stick to their places? and I was able to find a workaround for the problem.

Brief about the defect and solution

Moving the error message out of visible window area made it disappear. And user has to close the application ‘abruptly’. I was able to find a workaround for this without user closing the application.

Here goes the story in detail

I was able to reproduce the defect by following the steps provided by Amit (very well explained steps to reproduce), but I am not happy with closing the application ‘abruptly’. So started exploring (note exploring not googling) is there any option to get back the popup window. With my limited knowledge on Windows OS, I started with ALT-TAB option, but no use. Then I was thinking what can be done??? suddenly Mr. Task Manager  flashed in my mind. So I Right clicked on Task Bar to open Task Manager. Wow! The message pop-up window is back in the window pane. But wait!  I tried these steps again, but no use. So started investigating on the steps I tried

First Attempt:

1. Right clicked on Task bar

2. Clicked on Task Manager

Result:

1. Task Manager Window did not open

2. The Error message Pop Up reappeared in the window pane.

Second Attempt:

1. Right Clicked on Taskbar

2. Clicked on Task Manager

Result:

1. Task Manager Window opened up

2. But the error message Pop Up is not visible this time

This concluded that on first attempt I didn’t click on the Task Manager instead I clicked on some other options. So I looked into the nearest option available to Task Manager, and found that on clicking “Lock the Taskbar” option made the pop up to appear back. I didn’t stop here because now I am not sure whether the Lock up task bar was checked or unchecked initially. So tried all possible options with that and the conclusion is simple one liner “Toggle the Lock the Toolbar option to get back the pop-up to Visible window” 🙂

So to conclude, As a Tester

  1. I do a lot of exploratory testing
  2. I don’t stop just finding bugs, explore for reasons and also solutions
  3. I don’t Google as soon as I am struck; I explore various possible solutions before hitting Google. (Google might give the perfect solution at first page itself, if you are lucky. But what you might miss is hundreds of other valuable learnings, solutions and more importantly experience.)
  4. I do find solutions and bugs accidentally. Accidents don’t happen if you sit idle without exploring. (Wow! A new philosophy 🙂 )

How about you?

Read  http://www.satisfice.com/articles/et-article.pdf and http://testingeducation.org/a/nature.pdf to learn  more about ET.