As part of stuff that I’m working on, I’m surveying various tools to test new web applications that my group is working on. There are some interesting open source tools that can do some amazing things and I’m collecting a few of the links to docs and videos I’ve seen as part of my survey. I’ll update this post with info as I find it.
We (our customers) currently live in the IE 7/Firefox world with .NET and that is the focus of the tools I’m surveying.
For this effort, I define these testing types as:
Functionality Verification – Verifying the customer specified and engineering derived requirements for an application. This also includes acceptance testing for project/contract endpoints or milestones.
Regression Testing – During development and also when releasing new builds – verify that previously implemented functionality works as expected and that there aren’t unexpected side effects from the introduction of new functionality.
Stability Testing – Verification that an application will run without leaking memory, or bogging down both the client and server side and that there aren’t low probability catastrophic events that might bring an application down with no warning and limited visibility into what happened.
Load/Performance Testing – Assess how well an application runs on given hardware so that predictions can be made for the user experience. Establish scenarios of use for the performance testing. (How many times will someone ask – how many users does you app support? – this question doesn’t make sense to me unless you define what a user is doing and the scenarios for that user’s activity).
Selenium Core, Selenium RC, Selenium IDE, Selenium Grid
Note – Before I started this effort I did not realize how invested in Selenium Google is. Looking at the Google videos on YouTube made me realize that Google has hired the initial Selenium developer and is funding internal efforts to expand Selenium. Our use of Selenium can piggyback on this significant effort from Google.
This set of tools seems to be very popular and we’ve looked at using this combination on our app. Selenium in general supports all browsers since the actual operation is a core of Javascript. The IDE portion is enticing – in that it records (from Firefox) a user’s verbs on a web user interface. These actions are saved in "Selenese" – basically an html table that contains the actions done and also can include verification of things found in the resulting HTML coming back from the web server. These test can be placed in a menu of tests to run or can be run from an app using Selenium-RC – a java based testing exec service.
One drawback is that Ajax makes testing an app difficult since there isn’t a standard way to tell when the browser is done rendering your HTML that is shown in the browser. This makes Selenium-IDE difficult to use for recording a test – since you can’t instantly run the test again without a bunch of errors saying that UI elements are not present to be looked at or acted upon (at least for a while). Also – playing games with keystrokes are difficult to record from what the IDE records. You have to substitute recorded "typing" events in the Selenese so that your app is properly driven – creating Javascript events as normal.
Here are some things I’ve found that are useful:
Selenium: The in-browser acceptance testing tool
Jason Huggins at Google London Test Automation Conference (LTAC) giving a presentation of Selenium’s various components http://www.youtube.com/watch?v=78mts_sKNGs&NR=1 A build/verification environment is demo’d using various OS’s. A brief overview of Selenium-IDE and its relationship to Selenium-RC – in particular recording a test in Selenese and then automatically converting to one of various languages – Java, Ruby, Python, C#.
Selenium User Meetup 2008: Lightning Talks, March 4 2008
Google is funding Selenium development internally to support their Selenium test grid (Mark Striebeck – Google Test Manager).
Selenium Grid is new – supports multiple parallel test machines. End to end web testing is slow because of the comm between clients, web servers, and database. The presentation demo’d 20 Selenium tests running at one time. Any test that works currently for Selenium RC will work in the grid. Selenium Grid is available for download now.
Jennifer Bevan of Google shared experiences of the Selenium Grid work at Google. They have 51K tests per day on their Selenium Farm. Selenium has been adopted as the primary technology for functional testing of web applications within Google. The grid can error out due to Selenium RC bugs that effectively create a DOS on the backend test servers. There has been in house Google work to create utility functions to help out.
Google is feeding back updates to the open source Selenium group.
Selenium 1.0 is coming out – RC, Core, Grid and IDE together.
Selenium 1.1 – IDE enhanced to obey remote control instruction. Coming in a couple of months.
Selenium 2.0 – is going to leverage WebDriver.
GTAC 2007: Simon Stewart – Web Driver
Simon Steward presentation on the Google Channel. http://www.youtube.com/watch?v=tGu1ud7hk5I
Paraphased from the presentation:
"Speed is important"
"Driving a real browser in memory is just going to be slow".
Web Driver is very similar to Selenium RC and uses Java to create testing objects. A big difference from Selenium RC is that Javascript cannot be driven. WD uses direct API’s into the web browser as opposed to Javascript interfaces. This makes WD more efficient than Selenium RC. However, the lack of a Javascript interface makes testing for Ajax completions very difficult. We really need to know when the render is complete and our app has a Javascript flag to check for render complete. WD specifically does not have any mechanism to test for a render complete – you need to delay to handle these issues. This is really a killer for our app.
FAQ for webdriver: http://code.google.com/p/webdriver/wiki/FrequentlyAskedQuestions
GTAC 2007: Huggins & Stewart – Selenium-RC Vs WebDriver
Huggins & Stewart – Selenium-RC Vs WebDriver at The 2nd Annual Google Test Automation Conference (GTAC) in our New York office on August 23 and 24, 2007 http://www.youtube.com/watch?v=Vlz-WmcrBL8
Using open source tools for performance testing
Goranka Bjedov – Using open source tools for performance testing at Google London Test Automation Conference (LTAC) http://www.youtube.com/watch?v=KQ-wm3rW8_c&feature=related
Interesting point "what is a benchmark test – try to find out what customers’s are doing with the app in the real world, extract some subset of operations that reasonably reflect the system under real operating conditions and then every time code changes, subject the system to the test under load". Goranka suggest that engineering time be put into creating a benchmark test within the scripting language of the load tool of choice.
This quote puts our problems in perspective "At Google, at any point in time there are a thousand machines failing that need to come back in a reasonable amount of time". "Our systems are designed so that nobody should be able to see that".
About performance testing – "can our systems support what I think our customers would want".
Google’s requirement for user interface response time on search is that results are returned in less than a second and typically are in the ten’s of milliseconds.
When tests are monitored, a single resource will eventually be the bottleneck as simulated load is increased. Google recommended on Unix platforms, that the best customer experience is received when the limiting resource is not pushed above 80% of a maximum value. Google speculated that Microsoft’s limits are typically 60%. A problem for test engineers is that marketing groups will want the numbers on throughput when the limiting resource is 100%.
Goggle suggested tools:
JMeter Java Based
Grinder
OpenSta (Windows based)
General resources for open source testing:
http://opensourcetesting.org
http://testingreflections.org
Further reflections on this presentation from the following blog:
http://robertbaillie.blogspot.com/2006_09_01_robertbaillie_archive.html
Goranka Bjedov also posted an article about performance testing here:
http://googletesting.blogspot.com/2007/10/performance-testing.html