iTest

Last week I participated in a competition conducted by 99tests.com, a crowd sourced testing start-up from India.  This is  my approach\experience report on how I tested and won the competition.

What were the givens?

No requirement, no design documents, no test cases, no use cases, no user stories. All I had were credentials to login and the URL to test. So, do I need to agitate and not to start testing until I get requirements? Do I sit and write test cases? Any conventional scripted testers would’ve struck wondering what to do. So, here is a warning for all those scripted testers it looks things are changing because of agile approach and such crowd sourced testing services.

How did iStart?

I didn’t waste any time exploring the application. I thought the best approach with the givens (actually no givens ;-)) would be, jump straight into using the application and observing the behavior. I thought that would help with forming test ideas and then to build from there.

Being a user of such online shopping sites helped me to frame some initial expectation. So, started with Follow the (user) Flow heuristics. I decided to create my own credentials instead of using test credentials and login. Then observe how easy or buggy it is to find an item, add to cart, check out, paying through a third party payment gateway and choose a shipping address of my choice.

Did iFollow the flow?

Registration was successful  upon giving correct details. But I didn’t get to see any error message, so decided to enter some invalid data to find the application behavior. What I found in giving invalid input was the error messages were not user-friendly, not just in terms of the message, in terms of usability as well. Error messages were in a different page and users needing to click back button to get back to the registration page. On click on back button the entire data entered were lost. So, this made me take a deviation from my initial plans of Follow the flow. Decided to test this module thoroughly, wondering why? Read bug advocacy of this bug below.

How iDid bug advocacy?

I didn’t just log the bug with a summary line, description, steps to reproduce and screen shots. Instead, I also explained how and why this might bug the users and impact of the bug.

The user registration page plays a vital role in giving the first impression about the application behavior. Also, users use online shopping sites mainly to save time, by displaying errors in another page and asking the user to come back to previous screen to correct the data is actually wasting sufficient time. This advocacy helps in understanding the real impact of such bugs. So, always explain how any bugs\issues found would potentially affect the user experience.

Your bugs are your representatives. The bug logging also depends on context, if you know your developers very well and if he is sitting next to you, the bug logging, or advocacy may differ from the way you log bugs in crowd sourced testing. Here you have no clue about, who is doing bug triage, developers and their understanding of the product. So always give as much detail as possible so that they can’t reject your bugs.

Always remember this  most famous movie dialogs of all time, from “The God Father” movie while logging the bugs

“I’m going to make him an offer (details) he can’t refuse (to fix).”

Did iJust log bugs?

No, I went through most of the bugs logged by other testers. I posted comments, raised questions where ever I felt the bug was really not a bug or if the priority was inflated or if the issues were duplicated. Also, I neither missed to appreciate some good bugs reported by other testers nor missed to learn from fellow testers. I was actually a little disappointed that there were a significant amount of bugs without clear description and duplicates.

What were my objectives?

Than winning the competition, my objective was to log the maximum number of valid bugs. Ended logging 50 valid bugs, maximum by any tester. Also wanted to maintain a high bug acceptance ratio, 86% of the bugs were valid. Happy with that but still wanted to improve on acceptance ratio.

So, What iDidn’t test?

I found few even tested the Facebook ‘Like’ gadgets, Payment gateway and spell checks in the application’s blog. Though few of such bugs were valid, those were not from the application under test, so they got rejected. So,

Know your boundaries, so you do good enough testing in given time.

Ready to try some bug advocacy?

Here are a few other bugs logged by me, try to advocate for them. The application under test was an online shopping site.

  • “Similar Items” feature is missing.
  • Same book title, but displayed with huge difference in price tag.
  • The amount should always be right justified.
  • On entering the special characters the system through an “invalid gift message”
  • Pre order items are shown as Available and Buy Now.
  • Is the final price displayed in product description inclusive of taxes?
  • Can’t store search results.

Happy Bug Hunting!

P.S.

Some of my favorite lessons in Bug Advocacy chapter of “Lessons Learned in Software Testing”  book

55: You are what you write.

58: Your bug report is your representative.

83: The summary line is the most important line in the bug report.

89: Use market or support data when appropriate.

101: When you decide to fight, decide to win!

Advertisements

Be Prepared…

My previous post Focus on Defocusing ended saying “Problems unfold over time in ways that are often not predictable”. That is the fourth principle of  Context Driven Testing. You can never avoid problems, all you can do is be prepared to face unpredictable problems. Similarly you must be prepared to grab the opportunities that knock the door that are often not predictable.

How to prepare oneself for such situations?

Let us look back at the Journey of A R Rahman and learn from the legend how to prepare oneself for such situations. He lost his father at very young age,Since the pressure of supporting his family fell on him, he joined Ilayaraja‘s troupe as a keyboard player at the age of 11 and dropped out of school. Then he composed jingles for advertisements , later he moved into film music by Mani Ratnam. The salary he got for composing music for his first film Roja was 25,000INR. He would have earned this in a day or two by composing few jingles. He never thought about film music by then, it was his passion towards music and the opportunity, to perform and experiment at different level, were the reasons for accepting the offer. Well, the rest is History and  all knew his journey from then on and winning the Oscars in Feb 2009. Wait, did I say we all knew his journey? No, most of us do not know about the journey.

A R Rahman never agrees to all the offers. He always has a time slot left free to grab any great opportunity that comes up unpredictably.  The clear balance between the business motto and purpose motto helped to grab the Oscar opportunity. When Danny Boyle contacted him to compose music for Slum Dog Millionaire movie, he was able to grab that because he was prepared for such situations and he was ready to sacrifice a few business motto. If he was just having business motto he would have agreed for few other usual films and wouldn’t have time to accept this golden chance. If you notice his purpose motto of achieving something better and preparation were the keys for winning the Oscars. And now Mozart of Madras journey to conquer the world music starts with Journey Home World Tour 2010 today and this post is to celebrate that :), wait the journey didn’t start today, it stared at his very young age.

The purpose motto gave him better returns than what his business motto would have paid. So I strongly believe for both individuals and for companies, the key to success is

Striking a correct balance between business motto and purpose motto.

As an individual you should have a clear vision about your purpose motto. Don’t just run behind salary and the job that doesn’t satisfy you. In James Bach’s words

Reputation = Money. Build and protect your reputation.

Similarly companies achieve nothing great by just pressurizing employees with too narrow measurements of productivity and maximum utilization. If companies allow correct autonomy to employees, I am sure they will come with some creative ideas and that will help to achieve better returns than what could be achieved by focusing only on business motto. The number of Oscars received by ARR can be counted, but can never count how many were missed by others, who were not ready for such situations. Similarly the ideas implemented can be counted but never knew how many were missed because of running behind business motto.

Here are few pointers that will help you to get prepared yourself as a better tester Skilled Investigator to handle unpredictable problems and grab wonderful opportunities:

1.    Practice, practice, practice at Weekendtesting.

2.    Debate with people who have contrasting ideas. Note: Debate against ideas not against individuals.

3.    Always question useless practices, I strongly believe questioning leads to progress and questioning is not an easy to achieve skill, this again needs practice.

4.    Wait why am I typing all these? Am I going to offer better ideas than Michael Bolton’s? No. So why don’t you read Michael’s wonderful post and start Preparing.

Happy Preparation!

— A R Rahman Fan Devotee.

Note: All the incidents of ARR narrated here were from my memories, those were read over a period from various authentic magazines, articles and interview by ARR and\or his close associates.  So these may not be cent percent correct.

_________________________________________________________________________

Dear Rahman,

Yes,   your music wakes me up, makes me to think and heals me .You and Your music keeps inspiring me. I enjoy  listening to different types of music depending upon my mood. But only you and your music has the power to change my mood. Thank You Once again for all your wonderful music.

Share