Thursday, November 6, 2008

FIT mode best choice: running Selenium with Firefox Chrome TestRunner

I don't think Selenium-RC is the best choice for QA who does feature testing & regression testing with blackbox style, esp. those who doesn't have programming background.

S-RC(JAVA) disadvantages for blackbox QA/regression testing:
1) programming skills trainning cost for QA
2) test code complexity for maintenance
3) bad HTTPS/SSL page support
4) popup/alerts make automation stuck, require an autoclick alerts mechanism
5) not stable and with weird issues caused by Proxy Server
6) not lightweight tools, need setup environment
7) still need do 2nd around development to integrate with regression distributed system(even if S-GRID).

FIT mode is familiared by QA. There has 3 ways:
1) deploy Selenium CORE/TEST on testing server
2) MS IEHTA with test case repository anywhere
3) Firefox Chrome by Selenium-IDE with test case repository anywhere

For 1), some company has restriction to deny to do deployment
For 2), IEHTA is unstable and has critial issues such as multiwindow popup,xpath slowness etc. It's also hard to provide a session/cache isolation solution for multiple instances running simultaniously on one machine. It's also hard to create a recorder for IE.

I think running Selenium with Firefox Chrome TestRunner is the best choice for FIT mode.
1) Selenium-IDE is a good tool with recorder and still can do huge improvements.
2) Stable and well support HTTPS. Well support multiwindow.
3) multiple selenium running simultaniously on same machine with session/cache isolation by Firefox profiles mechanism
4) set FF preferences to let Selenium not stuck(click out popup/alerts)
5) save run snapshot/results/log to local file system other than postUrl solution
6) No need any proxy server. No need any additional installation and configuration.
7) Opensource Firefox platform is developer friendly. Firefox Chrome's XPCOM APIs can support us to implement all kinds of selenium features.

However, the bottleneck for openqa's FIT mode(Esp. Selenium-IDE) not become enterprise usage:
1) Selenium-IDE doesn't save down detailed enough snapshot/results/log to local file system which is the significant requirement for enterprise tools. Many other significant requirements also haven't been implemented.
2) FIT mode doesn't provide neccessary development SDK. We can build our enterprise test case maintenance system with GUI/webflow, parallel execution system with distributed infrastructure, result reporting system with DB/file system. However, we need SDK APIs to integrate Selenium FIT engine into these systems.

Most of developers are contributing for S-RC product on community. I hope openqa can also push Selenium-IDE FIT product line for enterprise usage.

No comments:

Post a Comment