Tag Archives: Selenium

Automated Testing, Selenium, Selenium IDE, Selenium RC, Software Development Tips, Software Testing, Windoes 2003 Server

Running Selenium RC Automated Tests Automatically Without a Windows Service

6 Comments

We have a suite of regression tests that we wanted to run automatically after our MSbuild build is complete. I’ve implemented these tests as C# code exported from Selenium IDE and preprocessed by some .NET code that I wrote. The tests are then compiled using MSBuild and executed.  The problem I ran into that wasn’t clearly documented anywhere was how to run a Selenium RC test as part of a build process – which requires the Java based Selenium RC server running on port 4444. Since we run our test as part of a scheduled task, how do we maintain the Java Selenium server running across logins of the scheduled task?

My first instinct was to run the Selenium server as a Windows service and I found a forum post that provided a .NET service to do this. The problem is that when the service runs in the LocalSystem’s account context, IE was problematic for enabling the proxy that Selenium RC uses during the invocation of tests.  Trying to modify the service to run as a legitimate user also didn’t work – since the service was running in a different Windows Station and IE did not run correctly in this mode either. 

It turned out the simple solution was to just invoke the Selenium Server from the build batch file and then terminate the server when I was finished with the test. There is a built in command in the command line interpreter named "taskkill" that does the job for me. Here is the .bat file code that I used – and note that the first line kills any java apps that are running so be careful.  I needed this line to abort a server in progress if the script or tests failed and left a java session hanging. My build machine doesn’t have any other java things going on so this worked for me. I’m using nunit to invoke my C# tests contained in "jalisngtests.dll". I’m also using the "sleep" command from the Windows 2003 Resource Kit.

taskkill /f /im java.exe
sleep 5
start "Selenium Server" java ^
-jar C:\selenium-remote-control-1.0-beta-1\^
selenium-server-1.0-beta-1\selenium-server.jar
copy .\bin\debug\*.dll
sleep 20
nunit-console jalisngtests.dll
taskkill /f /fi "WINDOWTITLE eq Selenium Server"
Automated Testing, JMeter, Selenium, Software Development Tips, Software Testing

Found This Blog Entry: Agile Ajax: Ajax Testing: Doubling Down with Selenium and JMeter

1 Comment

Agile Ajax: Ajax Testing: Doubling Down with Selenium and JMeter

This could be a solution for our automated testing issues.  Use Selenium IDE to record and test and then play back the test while JMeter is watching your HTTP traffic.

Automated Testing, Selenium, Selenium IDE, Software Development Tips, Software Testing

Our Problem with the Selenium IDE and AJAX

1 Comment

We have an application that uses quite a bit of Ajax and this caused problems in recording tests using the Selenium IDE. Each test step was recorded successfully, but because there are web services calls firing in the background on the client (the Ajax stuff), a playback would fail immediately. The fix in general is to use Javascript to check for a completion flag of some sort, but the IDE did not have a provision for inserting code automatically or to delay each test step.  This limited the ability to rerun a recorded test when mistakes are made during the recording process. I’ve been working (with other engineers) on modifying the Selenium IDE code to allow the automatic insertion of Ajax rendering complete Javascript code. However, the latest version of the IDE in SVN has a slow-down control now and that seems to work like a champ on our app.  It allows you to slow the playback down to something that approximates human speed and our app now seems to be happy as we record and then play back.  Thanks to the Selenium IDE developer Shinya Kasatani for this great addition.

I still plan to add a switch to allow specified Javascript to be executed automatically when the Selense plays back – to allow a check for a "render complete" flag that we have in our Javascript.

Automated Testing, Selenium, Selenium IDE, Selenium RC, Software Development Tips, Software Testing

Functionality Verification, Regression, Stability, and Load Performance Testing of Ajax Web Sites

1 Comment

As part of stuff that I’m working on, I’m surveying various tools to test new web applications that my group is working on. There are some interesting open source tools that can do some amazing things and I’m collecting a few of the links to docs and videos I’ve seen as part of my survey. I’ll update this post with info as I find it.

We (our customers) currently live in the IE 7/Firefox world with .NET and that is the focus of the tools I’m surveying.

For this effort, I define these testing types as:

Functionality Verification – Verifying the customer specified and engineering derived requirements for an application. This also includes acceptance testing for project/contract endpoints or milestones.

Regression Testing – During development and also when releasing new builds – verify that previously implemented functionality works as expected and that there aren’t unexpected side effects from the introduction of new functionality.

Stability Testing – Verification that an application will run without leaking memory, or bogging down both the client and server side and that there aren’t low probability catastrophic events that might bring an application down with no warning and limited visibility into what happened.

Load/Performance Testing – Assess how well an application runs on given hardware so that predictions can be made for the user experience. Establish scenarios of use for the performance testing. (How many times will someone ask – how many users does you app support? – this question doesn’t make sense to me unless you define what a user is doing and the scenarios for that user’s activity).

Selenium Core, Selenium RC, Selenium IDE, Selenium Grid

Note – Before I started this effort I did not realize how invested in Selenium Google is. Looking at the Google videos on YouTube made me realize that Google has hired the initial Selenium developer and is funding internal efforts to expand Selenium. Our use of Selenium can piggyback on this significant effort from Google.

This set of tools seems to be very popular and we’ve looked at using this combination on our app. Selenium in general supports all browsers since the actual operation is a core of Javascript. The IDE portion is enticing – in that it records (from Firefox) a user’s verbs on a web user interface. These actions are saved in "Selenese" – basically an html table that contains the actions done and also can include verification of things found in the resulting HTML coming back from the web server. These test can be placed in a menu of tests to run or can be run from an app using Selenium-RC – a java based testing exec service.

One drawback is that Ajax makes testing an app difficult since there isn’t a standard way to tell when the browser is done rendering your HTML that is shown in the browser. This makes Selenium-IDE difficult to use for recording a test – since you can’t instantly run the test again without a bunch of errors saying that UI elements are not present to be looked at or acted upon (at least for a while). Also – playing games with keystrokes are difficult to record from what the IDE records.  You have to substitute recorded "typing" events in the Selenese so that your app is properly driven – creating Javascript events as normal.

Here are some things I’ve found that are useful:

Selenium: The in-browser acceptance testing tool

Jason Huggins at Google London Test Automation Conference (LTAC) giving a presentation of Selenium’s various components http://www.youtube.com/watch?v=78mts_sKNGs&NR=1 A build/verification environment is demo’d  using various OS’s. A brief overview of Selenium-IDE and its relationship to Selenium-RC – in particular recording a test in Selenese and then automatically converting to one of various languages – Java, Ruby, Python,  C#.

Selenium User Meetup 2008: Lightning Talks, March 4 2008

YouTube Video

Google is funding Selenium development internally to support their Selenium test grid (Mark Striebeck – Google Test Manager).
Selenium Grid is new – supports multiple parallel test machines. End to end web testing is slow because of the comm between clients, web servers, and database. The presentation demo’d 20 Selenium tests running at one time. Any test that works currently for Selenium RC will work in the grid. Selenium Grid is available for download now.

Jennifer Bevan of Google shared experiences of the Selenium Grid work at Google. They have 51K tests per day on their Selenium Farm. Selenium has been adopted as the primary technology for functional testing of web applications within Google. The grid can error out due to Selenium RC bugs that effectively create a DOS on the backend test servers. There has been in house Google work to create utility functions to help out.

Google is feeding back updates to the open source Selenium group.

Selenium 1.0 is coming out – RC, Core, Grid and IDE together.

Selenium 1.1 – IDE enhanced to obey remote control instruction. Coming in a couple of months.

Selenium 2.0 – is going to leverage WebDriver.

GTAC 2007: Simon Stewart – Web Driver

Simon Steward presentation on the Google Channel. http://www.youtube.com/watch?v=tGu1ud7hk5I

Paraphased from the presentation:

"Speed is important"

"Driving a real browser in memory is just going to be slow".
Web Driver is very similar to Selenium RC and uses Java to create testing objects. A big difference from Selenium RC is that Javascript cannot be driven. WD uses direct API’s into the web browser as opposed to Javascript interfaces. This makes WD more efficient than Selenium RC. However, the lack of a Javascript interface makes testing for Ajax completions very difficult. We really need to know when the render is complete and our app has a Javascript flag to check for render complete. WD specifically does not have any mechanism to test for a render complete – you need to delay to handle these issues.  This is really a killer for our app.

FAQ for webdriver: http://code.google.com/p/webdriver/wiki/FrequentlyAskedQuestions

GTAC 2007: Huggins & Stewart – Selenium-RC Vs WebDriver

Huggins & Stewart – Selenium-RC Vs WebDriver at The 2nd Annual Google Test Automation Conference (GTAC) in our New York office on August 23 and 24, 2007 http://www.youtube.com/watch?v=Vlz-WmcrBL8

Using open source tools for performance testing

Goranka Bjedov – Using open source tools for performance testing at Google London Test Automation Conference (LTAC) http://www.youtube.com/watch?v=KQ-wm3rW8_c&feature=related
Interesting point "what is a benchmark test – try to find out what customers’s are doing with the app in the real world, extract some subset of operations  that reasonably reflect the system under real operating conditions and then every time code changes, subject the system to the test under load". Goranka suggest that engineering time be put into creating a benchmark test within the scripting language of the load tool of choice.

This quote puts our problems in perspective "At Google, at any point in time there are a thousand  machines failing that need to come back in a reasonable amount of time". "Our systems are designed so that nobody should be able to see that".
About performance testing – "can our systems support what I think our customers would want".

Google’s requirement for user interface response time on search is that results are returned in less than a second and typically are in the ten’s of milliseconds.
When tests are monitored, a single resource will eventually be the bottleneck as simulated load is increased. Google recommended on Unix platforms, that the best customer experience is received when the limiting resource is not pushed above 80% of a maximum value.  Google speculated that Microsoft’s limits are typically 60%. A problem for test engineers is that marketing groups will want the numbers on throughput when the limiting resource is 100%.

Goggle suggested tools:
JMeter Java Based
Grinder 
OpenSta (Windows based)

General resources for open source testing:
http://opensourcetesting.org
http://testingreflections.org

Further reflections on this presentation from the following blog:
http://robertbaillie.blogspot.com/2006_09_01_robertbaillie_archive.html
Goranka Bjedov also posted an article about performance testing here:
http://googletesting.blogspot.com/2007/10/performance-testing.html