HOLY COW!!!

HOLY COW!! It’s been long time since I blogged.

I was thinking for a while about what are the most important skills that help a tester to craft his own methods instead of following industry standards or best practices? This triggered while reading through one of the old posts of James Bach again recently,  especially this particular quote that grabbed my attention, “Following is for novices who are under active supervision. Instead, I craft methods on a project by project basis, and I encourage other people to do that..”

Heuristics

Extensive research is never possible in testing commercial software, due to time and budget constraints. So, how to test software quickly? Heuristics are used to speed up the process of finding an optimal solution. It’s something that helps you approach a problem and take decisions, which should be applied wisely. In order to write better on heuristics I wanted to know the Tamil (my mother tongue) meaning for heuristics. I decided to search in Google, from my knowledge and experience I believe Google would help me to get Tamil meaning of heuristics, but unfortunately it couldn’t get an answer for that. So, I decided to ask in Twitter addressing to few Tamil experts. This is heuristics, an approach to problem, but no guarantee that it solves problem.

Oracles

If heuristics help in approaching problems or take decision, Oracles help to identify problems. Some of  oracles used in testing are

1. Requirements

2. Comparing against competitors products

3. Consistency with in the products

L

“A tester is someone who knows that things can be different.” – Jerry Weinberg.

Y- Why?

“Questioning ignites thinking, leads to progress”  – Dhanasekar

Whenever I train novice testers, I provide them with few tasks, when they come back with a solution, I probe them with many questions like why was this done? why was this necessary? why do they think this is right solution? why not do it in a different way? Most times there won’t be concrete answers. My next step is to ask them to find answers for my questions. After finding answers  they would realise there were many more to learn and explore, but still many wouldn’t realise that those questions ignited their minds. The important thing is not to stop questioning. What we observe is not testing in itself, but testing exposed to our way of questioning.

Cognitive

Observation

If HOLY Sea(C) is preparation and practice for a battle, observation is the real battle. For instance, if your application sends emails, have you ever observed how it appears in the email alert message pop-ups in  many popular clients on windows platform or popular notifiers like Growl in Mac OS X? This example is out of an observation made on such email pop-up. This client we worked for, put their branding advertisements at the start of the email, so the alert pop-ups showed the first two lines of the content which is purely marketing text, but not important content. Because of which user might assume this to be spam and ignore it. This test idea would have never triggered if I missed to observe the alert pop-up. Making observations after executing tests is not end of testing, subsequent test ideas mostly result out of those observations. So, missing critical observation is missing potentially risky tests.

Advertisements

Test[ing this] Case

The first thing flashes in most software testers mind immediately on hearing software testing is Test Case. Even most of us explain our work to others and even to fellow testers something like this “I understand requirements, write test cases and test the application, log the bugs and mark the test case pass or fail based on the expected results”. I too started my testing career by explaining what a test case is and SDLC model that was (never) followed to an interviewer. But soon, I realized that it is practically not possible (or I am not intelligent enough) to write most test ideas (cases) even with detailed requirements. Most test ideas don’t popup while analyzing the requirements, but while performing the real testing on the product. The test cases written with requirement documents are only based initial abstract understandings made on theory, but the real practical ideas come while testing. But even recently many people advice to write detailed test cases, that even a layperson can understand. Are they recruiting a layperson or a smart tester? Why still the main focus is with writing detailed test cases even in agile world? Looking back the water fall model, what I can think of reason for writing detailed test cases is, there was  a phase in SDLC were developers are busy writing and developing a product for months together, before delivering into testing team. So, probably some smart manager got this idea of test case writing to keep the testers engaged during that phase. Few rituals still followed without analyzing, if those suit the context. 1.  Write detailed test cases for the projects where there is good clarity on requirements. We got clear requirements so write detailed test cases. 2.  Write detailed test cases where there are no clear requirements . The requirements are not clear let use have detailed test case document. 3.  Even if the project has very tight deadline, a certain percentage of time is allocated for test case writing instead of focusing on real testing . Because Test case documents are (mis)believed to be the only Savior of testers. 4.  I’ve even seen reverse engineering of coming up with test cases from a well functioning application, with the help of help files 🙂 5.  There is something called traceability matrix, which traces all the detailed test cases back to your requirement, I think this is designed especially to say the whole world, see we’ve spent so much of man power to duplicate the requirements and here is the proof for that 6. And test cases are again considered as Savior in this job hopping era, but reality is forcing to write such detailed test cases and asking the testers to blindly execute just the documented test cases too are the also one of the reasons for job hopping. Why a similar kind of test cases is demanded everywhere for every context?Why do test case document looks similar where ever I go? 😦 . The only standard that is followed more or less correctly in testing world is test case template. The problem with such detailed test cases writing and following them is, soon the application will develop resistance against bug, because testers were focused only on what is written in the test case nothing less nothing more. Any tests executed blindly by a human just against expected result is as good as (as bad as) automated test checks. Such practices never help to spot the Black Swan* [The disproportionate role of high-impact, hard to predict, and rare events that are beyond the realm of normal expectations] you have to explore more with real application to spot the Black Swan. Testing according to James Bach Testing is the infinite process of comparing the invisible to the ambiguous so as to avoid the unthinkable happening to the anonymous. So, let me try to form a different Black Swam theory, as a tester our job is to predict and spot the Black Swan before the customers or client spot it. 🙂 Even writing test cases need exploration, but it becomes a concern when the explorations stop (or forced to stop) once test case are written. So, constantly come up with new ideas to add more value. Instead of wasting time in writing detailed test cases cheat sheets like this  Test Heuristics Cheat Sheet would cover most of the basic test cases, why to write again? If you feel You Are Not Done Yet, then check this You Are Not Done Yet. And choose SBTM or any relevant format that suits the context. Any available basic requirement document plus the above cheat sheet should be more than enough to do a good enough testing.

Testers aren’t  for documenting, they are for providing valuable information.

P.S: Can you recollect  the color of each letter of Google logo in its home page? (Of course with no doodles), the page most of  us visit every second hour. If you can’t, then you are affected with major Black Swan blindness.

Happy testing your Case!

Dhanasekar S.

* I’ve just started reading The Black Swan book by Nassim Nicholas Taleb. Yet to read it completely.

Configure, Operate, Observe and Analyze

In one of my previous post I wrote about the below Windows  OS (XP with SP2) error message. This is one of very interesting behaviors that took me few hours to crack down.

As a novice Window’s(OS) user.

Access is denied.

Ah!  I am the only user of my machine, and I am the one who created this folder, no one else has any rights to restrict me from accessing my own folder.

Make sure the disk is not full or write-protected.

What the hell! Don’t say all this nonsense. I didn’t protect it from writing. I can write whatever I want in the disk. How do I know the disk is full? How do I write-unprotect? Or an unwrite-protect 😉 ?

That the file is not currently in use

No, I am not renaming any file. I am renaming a folder, what is your problem?

User annoyed and calls customer care.

Now, as an advanced computer user.

Access is denied.

Looks like the permission settings for the folder is changed, may be my wife? Son? Or Pet? Possible. Let me see the permission setting for the folder. Oh! This user has full access. Then, what is the problem?

Make sure the disk is not full or write-protected.

Nope, my drive has enough space, not write protected.

That the file is not currently in use

Files? I am renaming a folder. Let me see if the folder is kept open. Yes, so let me close and try. Still I am getting the same error.

Annoyed user might solve the problem by logging off or closing all files and folder or restarting the machine, which forces him to close all the programs kept open.

As a testers, start analyzing (Step 4 of testing Analyze, Step 1: Configure, Step 2: Operate, Step 3: Observe) the error message and follow the error message to see, to find the problem area.

Access is denied.

Does the application provide reasons to the Error? Yes it is it says

Make sure the disk is not full or write-protected.

The basic disc properties say it is either full or write-protected.

That the file is not currently in use

No, I am renaming a folder, but still let me check if the folder is  kept open? Yes.

So, let me close the folder and rename it. Even after that I am getting the same message.

Are there any files inside the folder?

Yes, there are few.

Are they open?

Yes, a word document inside that folder is kept open.

Is that causing the problem? Let me close it and try.

At last I can rename it, so “When a file inside a folder to be renamed is kept open, the user is not show with an appropriate error message” the problem summary?

I have just solved the problem but haven’t got the exact steps to reproduce.

The problem was identified without knowing the proper base step up or configuration. So let me go to base state, also let me note down or take a snap shot of the current state where the problem was first spotted, including other files\folders those kept open, processes running to name a few.

So, what is the base state? I may even restart my machine or log off or close all opened files and folders.

Step 1: Configure

Let me log off and log in. Note down the entire process running (and use some tools like Systracer that takes a snap shot of your machine)

Step 2: Operate

1.    Open the Disk Drive (Say D :\)

2.    Open the folder (Say D:\Test)

3.    Open a file inside the folder (Say D:\Test\test.doc)

4.    Use back button to get back to disk drive

5.    Try Rename the folder

Step 3: Observe

Ah! This time I am getting a proper message.

So, what could be the problem?

Let me try changing the sequence, let me do this

1.    Open the Disk Drive (Say D :\)

2.    Open the folder.(Say D:\Test)

3.    Open a file inside the folder. (Say D:\Test\test.doc)

4.    Close the folder

5.    Reopen the folder

Now the sequence of opened windows changed, now file first then the folder.

Nope! I got proper message. Tried with different types of file kept open like excel, txt files, folder within folders and also be keeping few more similar file types open in same\different folders.

With no success decide to get back to the dirty environment and try again, I opened few other windows ,files, browsers and application those were kept open when I first found this issue, and tried all the above steps, this time I noticed, I was  able to reproduce the error once in a while but not consistently. But after more than an hour I was not able to find the exact steps or pattern to reproduce it.

Then decided to quit and try later (Plunge and quit), but that one hour of exploration gave me confidence that it can be reproduced.

Few days later with lots of windows and browsers kept open I tried again to reproduce this. Initial few attempts were not successful.

When I was about to rename the folder, I noticed one of my friends pinging me in IM, so I went to the browser and replied to him, then came back to folder window to rename the folder.

Whoopee! I got error message I am looking for this time. It looks if I navigate between different windows I might be able to reproduce, but even trying that didn’t reproduce the error consistently. What next?? Trying out few other ways and finally was able to narrow down the steps to reproduce it consistently.

And, Here are the steps to reproduce.

1.    Open the Disk Drive (Say D :\)

2.    Open the folder (Say D:\Test)

3.    Open a file inside the folder  (Say D:\Test\test.doc)

4.    Use back button to get back to disk drive

5. Wait (approximately) 10 seconds

6.    Try Rename the folder

7.    Bingo!

Note to Readers:

  • Heuristics applied to solve are of James Bach’s Learning Heuristics SACKED SCOWS

Scouting Obsessively – discover the sources and tool you need
Authentic Problems – engage the mind
Cognitive Savvy – work with the rhythms of the mind
Knowledge attracts Knowledge – the more I know the easier I learn
Experimentation – make learning vivid and direct
Disposable Time – lets me try new things (I tried this in my disposable time)
Stories – are how I make sense of things
Contrasting Ideas – lead to better ideas (Skepticism, Critical thinking, Lateral thinking, Systems thinking)
Other Minds – exercise my thinking and applaud my exploits
Words and Pictures – make a home for my thoughts
Systems Thinking – helps me tame complexity

Also, Procrastination , Plunge In and quit.

  • How to reproduce is solved. Now find why this happens?
  • Never let the user to get annoyed. As a tester, analyze every behavior of your application.
  • Read How to Investigate Intermittent Problems to learn the ability and the confidence to investigate an intermittent bug.
  • “There are no accidents” –Master Oogway, Kung Fu Panda(Movie)

FaceBook Logoff

I have this habit of logging into any shopping site like Flipkart with same ID, same time in two different browsers (Firefox and Chrome). With one of the sessions, I browse through the items, and other one to add to the shopping cart. This is very helpful to me, to do shopping faster.

I thought I would do the same with Facebook as well. I would use one of the sessions to read all the status updates and other session to chat or render the videos and watch, so I can browse through all the updates from friends in quick time. But I observed logging into second browser forces to log out from the previous browser session

Is this a problem?

I don’t think as a tester you can decide if this is a really a problem assuming either one of those scenarios as an industry standards ;-).Even if one of those scenarios is set as an industry standards it is not necessary to follow them blindly. So Face off 1 yourself as a user to analyze such scenarios.

We cannot think of everything before we start.

You don’t know what you want until you see it.

While shopping user (like me) has two sessions opened. I am done with my shopping .I do check out after adding all the items to the chart and make payment, check my mailing address,which  takes close to 5 to 10 minutes. And then I log out of this session but completely forgot about the other session opened in different browser, and if the user is using a shared computer which is very common in India. So, here it looks it is better to log off all the sessions in the machine if user log off from a session. But one way it is helpful to user if he can access more than one session at a time.

So, which is the correct behavior? It is always decided by what your users are up to. So face off like a user and analyze the observed behavior.

Even let say Flipkart decides to change this and allows only one user session similar to Facebook by maintaining the sessions in server instead of cookies. It should be communicated to user in a very positive manner by highlighting the security risk. Any change like this cannot go without educating the user. Users who are very used to such behavior may not accept those changes.

User satisfaction comes from handling change, not mitigating risk.

Wait. Did I say it is the user always? No(w), I am seeing some new terminology being used these days apart from user friendly, its advertiser friendly ;-). Sometimes you have to consider advertisers as well, so no harm in annoying user for some time.  There you should not think about user friendly close buttons.

Aware of your potential user\customer very well, to analyze what you observed is really a problem. None of the so called industry standards or testing techniques like orthogonal array, boundary value analyses or whatever it is, will help to capture or analyze such scenarios, and we can’t estimate for such analyses in advance. So

Apply your sapience and heuristics. Don’t follow rules.

“Hell, there are no rules here, we’re trying to accomplish something” ~ Thomas Edison

Note to Readers:

  • Here I have given just one possible scenario that I generally do as a user, I like you to bring\think in different perspective to analyses.
  • I had a vague idea about this log off session, was thinking on how to shape this in to an article. Finally this got this shape after  reading this wonderful post by Nathan Smith Estimation is bunk

1 –  Face off (movie) – A revolutionary medical technique allows an undercover agent to take the physical appearance of a major criminal and infiltrate his organization.