Category Archives: testing

What, us a good example?

18 Thursday Aug 2016

Posted by in testing

≈ 1 Comment

I was pleased to hear Alan and Brent respond to my post “The black box tester role may be fading away” in AB Testing – Episode 43 (starting at 9:52). Their primary response was to my claim:

“The project structures they describe seem to be on the leading edge of the future of the software testing role. In my limited view of the software industry, I don’t see many companies that are anywhere near Microsoft in their evolution.”

They don’t think they’re really on the leading edge. I suspected that they would protest, because it’s human nature – everything is relative. If the organization I’m working with has a long way to go to achieve something that theirs is already doing, and I feel that theirs is a good enough exemplar, I don’t need to seek out something that’s even better. But from their point of view, they want to improve with the help of others who have gone even further, so their sights are set on others who have gone beyond. I’ve seen many software organizations recently that are so far behind in their evolution that it’s easy for me to point to Microsoft (which granted, I’ve also found many reasons to malign) and say that they’re far ahead of the pack.

These organizations that are behind the curve are generally surviving, and their companies are usually still making a profit. I saw this many times in my consulting. They are more or less successful and often don’t feel a strong need to make significant changes. This is where the traditional approach to software testing will hang on for a long time and keep the black-box testing role alive. At least for me, though, these are often not desirable places to work, and they may eventually find that it’s difficult to hire talented testers. Note that I’m more concerned about some of the antiquated traditional practices like scripted manual testing than I am with the black-box tester role itself, for now.

By the way, I’m amused that Alan introduced me as “our good friend,” but then didn’t seem to know whether that was okay to say. I’ve found that calling someone a friend often makes it so. I owe you guys a hug.

 

The black box tester role may be fading away

16 Tuesday Aug 2016

Posted by in testing

≈ 3 Comments

Is the traditional software tester role fading away? A recent blog post from Cem Kaner helped to reinforce this idea for me. You’ll find his comments inside this long post, in the “2. Oracles are heuristic” section, under the heading “A supplementary video”: Updating to BBST 4.0: What Should Change. Incidentally, the topic of the whole post is updating a course on software testing, which I think I attended a very early variation of around the turn of the millennium.

Cem’s parenthetical notes on tester careers are interesting. He suggests that traditional black-box testing (whether exploratory or scripted) will give way to piecework, where a tester will be paid by the number of completed tests. I’ve seen this model already underway in outsourcing companies like Rainforest QA and its competitors, where the manually executed test step is the basic unit that you’re paying for. It’s much easier to see this happening for scripted tests than for exploratory tests, and I argued against using this kind of service when an executive asked me to consider using it, because I see little value in scripted manual tests.

You can imagine that this kind of piece work will not pay testers very well. Cem noted that he already sees a significant pay differential between black box testers who don’t do any programming and those who have jobs that require some kind of programming. He suggests a few skills like programming that testers could add to become more marketable. I have a few items of diversification I can point to, including programming and automation skills, though I don’t often focus on automation. I’m familiar with testing web apps and mobile apps (and even mobile web apps :). My experience with embedded systems often gets the attention of recruiters.

I’ve added a few items to my resume this year that I’m happy about. I took a training course and became a Certified Scrum Master so I could understand the leadership aspect of agile processes better. Perhaps the most promising, given the current job market for security, is the Certified Ethical Hacker course I took and passed the exam for. I know that passing a certification exam doesn’t really prove anything, but these particular certifications do seems to carry some weight that might help my career. Both are subject areas I already have experience in, and I was happy to round out my knowledge in the classes.

I’ve been following Microsoft employees Alan Page and Brent Jensen on their AB Testing podcast. They both have had fairly traditional testing roles, but now are in roles that seem to be much more future-proof. Brent is now a data scientist, specifically, Principal Data Scientist Manager. Alan, Principal Software Engineer, describes himself as a helper, doing the odd (but challenging) tasks that don’t easily fit the developer roles on his team, which is something that appeals to me. Both are still involved in the testing process. The project structures they describe seem to be on the leading edge of the future of the software testing role.

In my limited view of the software industry, I don’t see many companies that are anywhere near Microsoft in their evolution. If the traditional black-box tester role is fading, I think it’s going to happen very slowly. I think it will require a very broad view of the industry to track a slow evolution like this, and I’m curious if you’ve heard from anyone who is in a good position to see it.

A bit of advocacy helps to earn a bug bounty

17 Sunday Jul 2016

Posted by in security, testing

≈ Leave a comment

I have been working on honing my security testing skills. I asked Don Ankney‘s advice on how to do this, and one of his suggestions was to participate in bug bounty programs. Many companies encourage security researchers to report security vulnerabilities to them, and in some cases, they offer monetary rewards to the first person who reports each one.

My first bug bounty report for Instagram, which wasn’t accepted, was discussed here: “Username Enumeration – Ho, Hum!” This time, though, I was more successful. I found that none of Instagram’s cookies on its web interface had the “secure” flag set, including the session cookie that identifies a logged-in user. Like username enumeration, the secure flag on the cookies is another “ho, hum” thing often excluded from bug bounty programs. But the Facebook Bug Bounty Program (which also covers Instagram) doesn’t mention such an exclusion, so I decided to report the vulnerability.

I spent some time crafting an attack scenario. I found that the attack didn’t work if I used “instagram.com” instead of “www.instagram.com.” I found that if the insecure page http://instagram.com was in the browser cache, the browser used the cached page and then there was no vulnerability. And for reasons I haven’t figured out, I was not able to complete the attack successfully if the victim was using Firefox. I was able to prove that hijacking an Instagram session was a simple matter of setting just the captured sessionid cookie. This is the bug report I sent:

Description and Impact

The secure flag is not set on any of Instagram’s cookies, including sessionid. When a user with an active session types “www.instagram.com” in their browser to go to the site, they will first hit the insecure site and transmit all of their cookies in the clear. An attacker monitoring their network packets will be able to hijack their session easily. Assuming there is no need to send cookies in the clear at any point, this is easily fixed by setting the secure flag in the cookies.

Reproduction Instructions / Proof of Concept

I implemented a proof of concept using Safari 8.0.8 on Mac OS 10.10.5 and Chrome 49 on Windows Vista Home Basic for the victim. I haven’t been able to reproduce it yet with Firefox.

  1. Make sure you’re not logged in to Instagram. Clear the browser cache.
  2. Go to https://www.instagram.com.
  3. Click “Log in with Facebook”, and enter valid Facebook login credentials. This logs you in to Instagram.
  4. …an arbitrary amount of time may pass, as long as the Instagram session is still valid when continuing.
  5. Go to a public network that someone is snooping on.
  6. Open a tab in the same browser as before and go to http://www.instagram.com (not https). The sessionid cookie is sent in the clear and has been captured by the attacker. Even though the server returns a 301 redirect to a secure site, the cookie has already been sent in the clear.
  7. Attacker hijacks the Instagram session by setting the sessionid cookie in their browser.

I got a reply five days later, saying “This is currently intentional behavior in our product…” I wasn’t surprised that another “ho, hum” bug was rejected, but I was surprised that they considered it a feature. So I replied, saying that I intended to publicly disclose the issue (which is standard practice after the report is closed, whether fixed or not) and I asked for further information about how the site needs this behavior in order to function, to inform my continued testing. I call this sort of response my “Just one more thing” reply, inspired by the TV character Columbo. This sort of followup is routine for professional software testers, but I don’t know how many security penetration testers put bug advocacy skills to use.

The next reply came quickly, saying that though many people had already reported this issue, they would go ahead and discuss the issue with the product team and try to fix it. And lo and behold, about three weeks later, I got notice that the issue is resolved, and I was pleasantly surprised to hear that they offered to pay me a bug bounty. The reasoning was fascinating – the site previously used http (I’m not clear how long ago) and then later switched to https. All the previous reports about this issue had been when they used http, which is silly, since in that case the secure flag would render the cookies invisible to the server. This explains their earlier pat rejection of bug reports about the secure flag, even though that response had become obsolete with the change to https.

They determined that I was the first to report the vulnerability since they switched to https, and so I qualified for the bounty. I am impressed with the amount of care that Facebook/Instagram took in handling this report. I’m eager now to dig deeper and apply more of my bug advocacy skills if necessary.

 

 

Username Enumeration – Ho, Hum!

28 Tuesday Jun 2016

Posted by in security, testing

≈ 1 Comment

I used to think that checking for username enumeration vulnerabilities was important to do. Based on what I’ve observed, now I’m not so sure.

When I’m conducting a broad test of a software system, I tend to check for basic security holes like username enumeration. “Username enumeration” vulnerabilities happen when software that has login accounts for its users gives you a way to build a list of valid login accounts, for example, by giving you a way to query whether a guessed username is valid or not. Once you have this list, you can try to launch a brute force attack to guess the passwords.

Testing for username enumeration is one of the mitigations recommended in the 2013 OWASP Top 10. Cigital gives it even more prominence in their Top Web Application Security Vulnerabilities Compared to the OWASP Top 10, with username enumeration ranking as the 9th most commonly found, even when only considering the vulnerability via password reset features. My experience, though, is that companies aren’t very interested in fixing these vulnerabilities. It may be commonly found because it isn’t commonly fixed.

Once I did get a fix for an obvious vulnerability, but when I found I could still do enumeration by checking the response time from the server, that didn’t get fixed. I often see no fix at all when I report a username enumeration problem. For example, the main web login form for Instagram has this vulnerability, but Instagram considers your username to be public information, and they confirmed when I contacted them that they’re not going to close this hole. A significant fraction of the companies that participate in the HackerOne bug bounty program specifically state that they exclude username enumeration from the program.

I’ve found a few ways that companies have indirectly mitigated this issue, which may be contributing to some of the “ho-hum” response:

  1. Rate-limiting on the vulnerable feature based on IP address. In my testing, it wasn’t uncommon to see that a feature that allowed me to enumerate would temporarily lock me out after about several tries in rapid succession. This would only succeed in locking me out if it’s based on my network presence, not on the username, since I would be trying a different username each time.
  2. Similarly, a few vulnerable sites use CAPTCHA to defeat automated enumeration attempts. After trying several different usernames, I would get a CAPTCHA challenge that stopped a script from continuing.

In both of these cases, I can easily determine if a particular username is in use. But if I want to compile a large number of usernames, it may take a script months of running at the maximum allowed rate. There may be other ways to defeat these measures, such as frequently changing the public-facing IP address, using a botnet, or trying to use a database of known CAPTCHA responses.

None of this matters for Instagram, because the vulnerability on the web login is not rate-limited in any way. And that isn’t even the easiest vector – the issue with incremental userIDs mentioned here has not been fixed: InstaBrute: Two Ways to Brute-force Instagram Account Credentials.

One remaining mitigation comes to mind – the account lockout based on failed password attempts. Once we have a list of good accounts, we haven’t gained anything until we guess the password. I haven’t tried cracking that nut yet. My first thought is that if we have a large number of usernames, the time it takes to try one particular password for each of them may not trigger an account lockout at all by the time we roll back to the top of the list for the next password to try, as long as the lockout feature automatically resets itself after some fairly short period of time. But perhaps I’m being naïve about how quickly we would need to progress through a long list of possible passwords.

I would be curious to hear what your experience has been with reporting username enumeration problems, especially with companies that set a high priority on closing these holes.

Image credit: Christiaan Colen (CC BY-SA 2.0)

 

Better Bisecting

01 Wednesday Jun 2016

Posted by in testing

≈ Leave a comment

Tags

Bisection is a procedure that software testers can use to isolate a defect. I’ve been having fun building a tool to assist with bisection, and I’m writing about it in order to get feedback on whether it may be useful.

This work was inspired by the “bisect up” and “bisect down” features that James Bach developed for the perlclip tool. These features are tied to the counterstring generator. You start by creating counterstrings of two different sizes (perhaps vastly different sizes), where generally you see that when you use the smaller of the two strings in a test, it passes, and you observe that a test with the larger string fails. The task is then to determine precisely where the boundary is between passing and failing. You can use the “u” (bisect up) and “d” (bisect down) commands depending on whether the last test passed or failed to generate additional test strings that bisect the remaining possibilities until you converge upon the boundary.

I’ve used this feature many times – it’s really helpful. But I often find that I get confused trying to keep track of whether I should go up or down. If I make a mistake, I have to back to the last two values I’m confident in, create two counterstrings that I don’t actually use, and start over bisecting from there.

Here is my redesign that I think makes bisection easier to do. This is implemented in testclip, my work-in-progress port/enhancement of perlclip using Ruby. I’m going to demonstrate the tool by finding a bug in Audacity (version 2.1.2 on Mac OS).

I created a new Audacity project and looked for a text field that would be a good candidate for long string testing. I clicked File > Edit Metadata and found what I needed. I fired up testclip and make a counterstring for a happy path test:

$ ./testclip.rb 
Ready to generate. Type "help" for help.

cs 10
counterstring 10 characters long loaded on the clipboard

Then I pasted the data into the first field on the Metadata Tags dialog in Audacity:

metadata10

I clicked “OK,” then opened the dialog again, and the data was preserved without any errors. So I established that the counterstring format is valid for this field (some input fields don’t allow asterisks or numbers, so it’s good to check). I recorded the result of the test in testclip:

pass
10 recorded as pass

That was boring. Next I wanted to try a large number. In my experience, 10,000 characters is very fast to generate, and larger than what most input fields need, so that usually where I start. I asked testclip for a 10,000 character string.

cs 10000
counterstring 10000 characters long loaded on the clipboard

This was the result of the test:

metadata10000

When I tried to move the cursor further to the right, the cursor moved off the window, and I couldn’t see any more of the string. Looks like I found a bug. I record the result in testclip, choosing the tag “obscured” to identify the particular failure. This may become important later if I find a different failure mode.

fail obscured
10000 recorded as fail obscured

Due to the nature of the counterstring, I already had it pretty well isolated – it’s likely that my boundary is between 5690 and 5691 characters. But let’s make sure. I generated a 5690-character counterstring, find that it works fine, and recorded that as a pass.

cs 5690
counterstring 5690 characters long loaded on the clipboard

pass
5690 recorded as pass

I can ask testclip to report the test results it knows about and automatically identify where the boundaries between different results are.

status
10: pass
5690: pass
--boundary 1
10000: fail obscured

Next I tried 5691. This failed as with the 10,000 character string. I recorded this as the same type of failure and show the status again, which shows that testclip puts 5691 and 10,000 in the same equivalence class, just as it did with 10 and 5690 (I’m hoping it’s not confusing to call this an “equivalence class”, which is a term usually used to describe expected results, not actual results).

cs 5691
counterstring 5691 characters long loaded on the clipboard

fail obscured
5691 recorded as fail obscured

status
10: pass
5690: pass
--boundary 1
5691: fail obscured
10000: fail obscured

So, I hadn’t needed to bisect anything yet. I decided to make the test more interesting by saving the project and opening it again, to see if the long string is saved and loaded properly. I went back to 5690 and did the save and load test. Note that generating a new counterstring would reset the bisection status in perlclip, but all the test results are still retained in testclip so I can track multiple failure points. And in fact, the test fails, because after I open the saved file, the field is completely empty.

So now I abuse the tool just a bit, changing the result of the 5690 test. I’m actually running a slightly different test now, but I think I can keep it all straight. I tag this new failure “empty”. I now have two boundaries:

cs 5690
counterstring 5690 characters long loaded on the clipboard

fail empty
5690 result changed from pass to fail empty

status
10: pass
--boundary 1
5690: fail empty
--boundary 2
5691: fail obscured
10000: fail obscured

I have no clue where the new boundary 1 lies, so I’ll use bisection to find it:

bisect 1
highest value for 'pass': 10
lowest value for 'fail empty': 5690
2850 characters loaded on the clipboard

This test also failed, so I recorded the result and bisect again.

fail empty
2850 recorded as fail empty

bisect 1
highest value for 'pass': 10
lowest value for 'fail empty': 2850
1430 characters loaded on the clipboard

To complete the bisection, I repeated this process: do the next test, record the result, and bisect again. This was the end result:

bisect 1
Boundary found!
highest value for 'pass': 1024
lowest value for 'fail empty': 1025

status
10: pass
720: pass
897: pass
986: pass
1008: pass
1019: pass
1024: pass
--boundary 1
1025: fail empty
1027: fail empty
1030: fail empty
1075: fail empty
1430: fail empty
2850: fail empty
5690: fail empty
--boundary 2
5691: fail obscured
10000: fail obscured

I reported the two bugs to the Audacity project, and called it a success. Note: this was a case of a moving boundary – on two previous days of testing, the failure point was at 1028 and 1299, though within each test session I didn’t observe the boundary moving.

Besides adding the rest of the perlclip features and porting to Windows, the next things I’d like to implement for testclip are snapping the bisection to powers of 2 and multiples of 10, since bugs tend to lurk in those spots, and finishing a feature that can bisect on integer values in addition to counterstrings.

For more about perlclip, see “A Look at PerlClip” (free registration required).

photo credit: Wajahat Mahmood

Amplifying the Comment Challenge

31 Tuesday May 2016

Posted by in testing

≈ 1 Comment

Tags

An ongoing topic on Twitter is resonating with me – the #CommentChallenge, described in this blog post by Kristīne Corbus – Comment Challenge. The challenge is basically to leave a comment on at least one blog post a week that’s relevant to your work.

This is something I tend to do anyway. It doesn’t matter if I’m reading an article, book, discussion forum, or blog post, or listening to a podcast or any other media, I’m always looking for a way to engage with the author. If an author gives me something useful, I have an opportunity to make it a richer experience with the author’s help, and possibly strengthen my network. Also, if I expect I may be posting a comment, I find that I read the information more carefully and I’m therefore more likely to retain it.

When I’m reading a blog, the form of my feedback may be a blog comment, but that’s not generally a great platform for an extended discussion. So if I really want to get into the topic and I know where the author hangs out online, I may start a discussion elsewhere. I’ll probably also post some sort of comment on the blog, because that helps to show the public that the blog has engaged readers (especially if there are no comments yet), which helps the author. A habit that serves me well is just trying to be helpful.

These are the types of situations where I tend to offer comments:

  • When I have a question about something the author said or a closely related topic, something I’d really like to learn – either the author’s opinion, or facts that are hard to find elsewhere.
  • When the author asked a question and I have potentially a useful answer.
  • When I have something to add to what the author said that I think will be highly valuable, even if the author didn’t ask for this type of feedback. I try not to do this very often.
  • When I disagree with something the author said. I think carefully before I do this. Doing this can often earn the respect of the author, and we’re both likely to learn something from the exchange, but when done in the wrong way, it can damage both my relationship with the author and my public reputation. I won’t try to elaborate on all the subtleties here. Often I just ask a question instead of stating directly that the author is wrong. I may find out that I misunderstood something they said, and I don’t actually disagree with them.
  • When I want to give kudos to the author for making an important point or for making a point particularly well. This type of feedback is less useful than the rest, so it’s best to combine it with one of the items above.

I still don’t comment on everything I read – only those things that I have a useful reaction to that I can share.

My challenge to myself is a bit different from the Comment Challenge. I tend to let my learning habit fizzle, so that I stop taking the time to read or otherwise learn new things. So my challenge is to expose myself to new things every week. When I do that, the comments will naturally follow.

The Ethics of Testing a Public Server

26 Thursday May 2016

Posted by in testing

≈ 2 Comments

I like to hone my testing skills by trying different techniques. Sometimes the project I happen to be working on serves well as a sandbox for this, but not always. I also like to write about testing techniques using examples that other people can try. So it’s convenient to have an easily accessible application that I can write about.

I’ve been working on generating test data like long strings and large numbers with the venerable perlclip tool and a partial perlclip port to Ruby that I call testclip. I’m curious what you think about the ethics of testing in each of these real situations below.

1) Sorry, Wikipedia

I was having a discussion with a contact at Wikipedia, and I wanted to illustrate how I use bisection with long strings to isolate a bug. I wanted to find a bug on Wikipedia itself, so I tested its search feature. I considered the risks of testing on their production system – though long strings are fairly likely to find a bug, I couldn’t remember ever seeing them cause a catastrophic failure. So I judged that it was appropriate to continue. I think my contact was aware that I was testing it, but I didn’t explain the risks and he didn’t grant explicit permission.

Wikipedia gave me an ideal example, with a minor failure on a moderately long search string, and a more severe error with a much longer string (I went up to about 10,000 characters). I started writing up my analysis. As I went back to reproduce a few of the failures again, I noticed a new failure mode I hadn’t noticed before. Rather than isolate this new failure, I decided to stop testing. It seemed unlikely that my testing was related to this, but I wanted to make sure.

When I got in touch with my contact at Wikipedia, I found out that I had caused a major worldwide outage in their search feature. I did a lot of reflection after that – I really regretted causing this damage to a production system.

Was it ethical for me to run these tests?

2) Please test my site

I listened in to the virtual STAR East 2016 conference, which had a Test Labs activity that was accessible for virtual participants. I didn’t really understand what the activity was, but I did see that we were invited to test a particular open source application, CrisisCheckin, and report bugs on GitHub. An instance of the server was set up for testing. I used this as motivation to add a feature to testclip to bisect on an integer value in addition to the length of a counterstring.

It was nice to have a test instance of the system. I still considered the possibility that my testing could cause an outage that would affect the other people who were using the test instance. I decided to take the risk. The long strings I tested with made all similar types of data slightly more difficult for all users to read on the page, and in some cases the user interface didn’t provide a way to delete the data, so I did have a small impact on the shared system. I didn’t cause any outages that I was aware of.

There were instructions on GitHub for setting up a local instance of the software, which would be ideal in terms of not interfering with anyone else’s use of the site, but I chose not to take the time to do that.

Would you agree that my testing in this case was ethical?

3) It’s popular, so I’m picking on it

I’m working on writing an example usage of perlclip now, where I chose to pick on the main Google search field. I tested with a search string up to 1000 characters long, which finds a minor bug, but doesn’t seem to affect the availability of the system.

Is it ethical for me to do this testing, and publish something that encourages others to do the same?

A common reaction to these questions I’ve heard is that it’s the responsibility of the owners of the web site to make the site robust, so it’s not my fault if I’m able to do something though the user interface that breaks it. I don’t think it’s that simple.

I perused the Code of Ethics for the Association for Software Testing, and I didn’t see anything that directly addresses this question, though it’s clear on what to do when we do cause harm. At least for example 1 and 3 here, I’m not using these services for the purposes they were intended for. The Terms of Service for Google don’t actually say that I have to use it for the intended purpose. The Wikipedia Terms of Use, though, do talk about testing directly, which is expressly allowed in some situations. This testing is not allowed if it would “…unduly abuse or disrupt our technical systems or networks.” The terms also don’t allow disrupting the site by “placing an undue burden on a Project website.” So clearly it’s bad to cause an outage, but difficult to assess the risk in advance of an outage happening.

It’s much more clear that it’s not okay to conduct security testing without explicit permission. Security testing includes looking for denial of service vulnerabilities. But my intentions for doing long string testing generally aren’t to find vectors for a denial of service attack, even if that’s what happened in one case.

So how much caution is warranted to mitigate the risks of long string testing on production servers?

If the conclusion is that we should never test with long strings in production (at least without permission), then we have to look for safe places to practice our testing skills. Running a personal instance of an application server is one option, but that isn’t easy for everyone to do. Another option is having a public sandbox that we can access, as we have with CrisisCheckin. There are several cases of servers set up for educational purposes, either associated with exercises in a book or with a training class. Many of those, though, are only intended for customers who bought the book or the class. I think I’ll shift my focus to native applications that run locally and are easy to install. My head is in the web so much, I forget that there is such a thing as a local application. 🙂

Podcasts I’m listening to

04 Wednesday May 2016

Posted by in testing

≈ 1 Comment

I’m going to start writing about software testing again; an easy way to jump in is to discuss podcasts. I’ve been listening to a lot of podcasts lately to hone my technical skills.

The podcast I’m most familiar with that’s directly related to software testing is AB Testing, hosted by Alan Page and Brent Jensen from Microsoft. These guys are right on the leading edge, and they go into a good amount of detail about how they do what they do. They aren’t afraid to express opinions about what they don’t like, too.

I’ve also recently started listening to TestTalks by Joe Colantonio and Testing In The Pub by Stephen Janaway and Dan Ashby.

On the development side, I like Developer on Fire, hosted by David Rael. David has amassed an impressive range of interviewees. I’ve also been listening to Agile for Humans, from Ryan Ripley, Don Gray, Tim Ottinger, Amitai Schlair, and Jason Tice.

I’m trying to become more knowledgeable about software security, with the help of the Silver Bullet Podcast from my former co-worker Gary McGraw.

What else can you add to the list?

 

Unit Testing – A Taste of My Own Medicine

10 Friday Nov 2006

Posted by in archive, technology, testing

≈ Leave a comment

Tags

, ,

Throughout my career, I’ve been fussing at software developers about not doing decent unit testing. Recently I had to focus that fussing on myself.

I like the idea of test-driven development, where an automated unit test is developed in conjunction with the product code. But I don’t do it very often. Sometimes it’s because I don’t know if I’ll ever use the code I’m writing again – it’s just a one-off thing that won’t need to be repeated after the code does its work once. Sometimes it’s because of the difficulty in setting up a unit test, like my code that uses Watir to automate Internet Explorer. Some of my excuses are the same kind of excuses that the developers I’ve been fussing at have used.

I’m not ready to draw any big conclusions about unit testing, either for my test automation or for product development, but I do want to relate a success story. I had been struggling for a couple of days with a Perl program that parses an SCL script (OpenSTA’s scripting language). It would read in the SCL file, piece together all the line continuations so I could search each statement as a single entity, modify some of the statements, then split them back up into shorter lines. It was mostly working, but little glitches kept appearing – extra blank lines would appear, a line would be cut off too short, etc. As soon as I fixed one glitch, another would appear – the parsing process was fairly complex.

Finally it dawned on me that if a developer were to describe this problem to me, I would suggest that he immediately write unit tests. So that’s exactly what I did. I tried Perl’s Test::Unit module in earnest for the first time. It took a while to figure out how to set up the tests – the examples in the documentation are too trivial to show how a real unit test would work. But after a few hours, I was adding tests, and the tests were finding bugs. It only took a handful of rounds of adding a test, fixing a bug, then finding and fixing the bugs that the fix uncovered. The glitches were gone.

So while I struggle to find the right balance to determine what test automation code should have unit tests, I’ve learned that the code that isn’t working is definitely a good candidate for unit testing.

Originally published on the Tejas Software Consulting Newsletter blog.