Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Journey from Selenium to Playwright for running more efficient automated tests on our Jira Add-on

As a proud Atlassian Marketplace Partner, Tappointment emphasizes thoroughness and efficiency in automated testing for our Jira add-ons. 

👩‍💻🔧 Discover the road our dedicated team of QA engineers took to choose Playwright as our new test framework.

One of the main benefits is the seamless integration with Jira, including its crucial support for iframes—an essential functionality for embedding our application in Jira.
🔑 But we also took into account other key factors that ensure the reliability and efficiency of our testing process.

  • Streamlined setup without additional installations
  • Remarkable browser compatibility
  • User-friendly test runner UI
  • Integrated reporting features
  • Automatic waits for enhanced efficiency

Check the full post below. 

---

Playwright a web browser automation tool developed by Microsoft, designed to thoroughly test web applications from end to end. It supports multiple programming languages such as Python, JavaScript, Java and C#, and allows testers to conduct their browser automation tests in Chromium, WebKit and Firefox. Playwright is also suitable for mobile testing on Chrome and Safari.

Importance of browser automation in software development

Every IT company needs to deliver high-quality products, so they need to test their system thoroughly. Beyond traditional manual testing, automation is vital in ensuring software quality. Browser automation plays a critical role in covering regression testing, UI/UX evaluations, and verifying cross-browser compatibility.

This approach alleviates the testing team's workload, optimizes efficiency, and conserves valuable time and resources through the automation of repetitive tasks. Moreover, integrating automation testing into Continuous Integration/Continuous Deployment (CI/CD) pipelines enables developers to immediately check their code and ensure that any introduced modifications do not break existing features.

Early testing is crucial to avoid prolonged debugging sessions, project delays, and low-quality product releases. 

Test automation frameworks

At Tappointment, when we first had the opportunity to write automated tests, we were debating which framework to use. 

Initially, we found two worthy candidates: Cypress and Selenium. Both browser automation tools have their strengths and weaknesses, but after weighing the pros and the cons, we decided on Selenium to automate the regression tests of our popular Jira retrospective tool >  Power Retro | Agile Retrospective for Jira ✨ with AI ✨

Road to Playwright 

Our initial choice of Selenium was driven by its capacity to seamlessly integrate with iframes, a crucial functionality required as Jira exclusively provides an iframe for embedding our application. Cypress, at the time, did not offer comparable support for iframes. Additionally, Selenium's broad browser support, encompassing Chrome, Firefox, and Safari, aligned with the primary focus of our company's projects. The framework's popularity, coupled with our testers' proficiency in Python—a language fully supported by Selenium but not by Cypress—played a significant role in shaping our decision-making process.

Swiftly immersing ourselves in test case development, we used Selenium to craft approximately 14-15 tests. The integration of Selenium with our CI/CD pipeline in Gitlab proved instrumental, enhancing our ability to identify and address new bugs before deployment. As the product's development progressed, the necessity arose to adapt the tests to accommodate new features, prompting the occasional creation of additional tests to augment our regression test suites.

While Selenium initially met our requirements, a significant challenge emerged as the tests exhibited flakiness, frequently reporting false positives and false negatives. This drawback became counterproductive, consuming more time than manual testing following system changes. Recognizing the need for a more reliable solution, our quest led us to Playwright. 

Initial thoughts

Playwright received numerous positive feedback for its reliability, prompting me to explore its capabilities for our testing needs. Initially, I attempted to recreate our tests using Python, given its compatibility with Playwright. However, I encountered limitations as certain user-friendly features of Playwright were unavailable in the Python implementation. As a result, despite my limited proficiency in JavaScript and TypeScript, I opted to develop tests using TypeScript. The framework's ease of use became apparent, with any challenges arising primarily from my relative unfamiliarity with the programming language.

Key benefits

Installing Playwright is a straightforward process; it can be easily set up through npm without the need for additional installations, as opposed to Selenium.

Playwright boasts remarkable browser compatibility, eliminating concerns about code working seamlessly across various browsers like Safari and Chrome. In contrast to Selenium, where different drivers had to be created and installed for each browser, Playwright streamlines this process, requiring no extra packages for immediate functionality. This not only simplifies the code but also allows for the creation of separate browser contexts, facilitating parallel test execution.

The inclusion of Playwright's test runner UI enhances the testing experience. It simplifies test execution, debugging, code navigation, element inspection, console monitoring, and network analysis. Additionally, the UI provides an effortless way to view screenshots during test runs, all without the need for additional code. Configuration settings in the config file enable the framework to automatically save screenshots on failure and record execution videos.

Playwright's automatic waits offer a substantial advantage over Selenium, where extensive waits were necessary for proper functioning.

The reporting feature is integrated into Playwright, offering diverse output formats such as blob, xml, json, and html, as well as CLI options in both line and dot modes. Furthermore, users have the flexibility to implement custom reporting methods. This functionality proves especially beneficial when incorporating tests into CI/CD pipelines, particularly when there is a requirement to store reports within Jira.

Most notably, Playwright has proven to be highly reliable in our experiences so far. The singular drawback encountered pertains to the insufficient development of its documentation and a notably smaller community compared to Selenium.

Conclusions

Our transition from Selenium to Playwright at Tappointment has markedly enhanced our testing efficiency and reliability.        

While Selenium initially met our requirements, the streamlined setup, unparalleled browser compatibility, user-friendly test runner UI, and integrated reporting features of Playwright have proven instrumental in overcoming challenges and optimizing our testing processes.        

Despite encountering a limitation in Playwright's documentation and a smaller community, the benefits in terms of reliability and ease of use have decisively outweighed this singular drawback.        

In our commitment to delivering high-quality software products, we persistently explore optimal solutions for our automated testing requirements, with Playwright currently serving as our preferred framework.    

2 comments

Comment

Log in or Sign up to comment
Mark Thomas February 13, 2024

This was informative. Thanks for writing it up. I've heard good things about Playwright; I'll have to give it a try.

My QA team is already working on an in-depth research and experience update with the framework, will post it as soon they are publishing it. 

Ulrich Kuhnhardt _IzymesCo_
Marketplace Partner
Marketplace Partners provide apps and integrations available on the Atlassian Marketplace that extend the power of Atlassian products.
February 13, 2024

Couldn't agree more! Love playwright - and if you're stuck figuring out the right locator for that nested Atlaskit select - you can get unstuck with playwright codegen !

TAGS
AUG Leaders

Atlassian Community Events