As a proud Atlassian Marketplace Partner, Tappointment emphasizes thoroughness and efficiency in automated testing for our Jira add-ons.
👩💻🔧 Discover the road our dedicated team of QA engineers took to choose Playwright as our new test framework.
One of the main benefits is the seamless integration with Jira, including its crucial support for iframes—an essential functionality for embedding our application in Jira.
🔑 But we also took into account other key factors that ensure the reliability and efficiency of our testing process.
- Streamlined setup without additional installations
- Remarkable browser compatibility
- User-friendly test runner UI
- Integrated reporting features
- Automatic waits for enhanced efficiency
Check the full post below.
---
Playwright a web browser automation tool developed by Microsoft, designed to thoroughly test web applications from end to end. It supports multiple programming languages such as Python, JavaScript, Java and C#, and allows testers to conduct their browser automation tests in Chromium, WebKit and Firefox. Playwright is also suitable for mobile testing on Chrome and Safari.
Importance of browser automation in software development
Every IT company needs to deliver high-quality products, so they need to test their system thoroughly. Beyond traditional manual testing, automation is vital in ensuring software quality. Browser automation plays a critical role in covering regression testing, UI/UX evaluations, and verifying cross-browser compatibility.
This approach alleviates the testing team's workload, optimizes efficiency, and conserves valuable time and resources through the automation of repetitive tasks. Moreover, integrating automation testing into Continuous Integration/Continuous Deployment (CI/CD) pipelines enables developers to immediately check their code and ensure that any introduced modifications do not break existing features.
Early testing is crucial to avoid prolonged debugging sessions, project delays, and low-quality product releases.
Test automation frameworks
At Tappointment, when we first had the opportunity to write automated tests, we were debating which framework to use.
Initially, we found two worthy candidates: Cypress and Selenium. Both browser automation tools have their strengths and weaknesses, but after weighing the pros and the cons, we decided on Selenium to automate the regression tests of our popular Jira retrospective tool > Power Retro | Agile Retrospective for Jira ✨ with AI ✨
Road to Playwright
Our initial choice of Selenium was driven by its capacity to seamlessly integrate with iframes, a crucial functionality required as Jira exclusively provides an iframe for embedding our application. Cypress, at the time, did not offer comparable support for iframes. Additionally, Selenium's broad browser support, encompassing Chrome, Firefox, and Safari, aligned with the primary focus of our company's projects. The framework's popularity, coupled with our testers' proficiency in Python—a language fully supported by Selenium but not by Cypress—played a significant role in shaping our decision-making process.
Swiftly immersing ourselves in test case development, we used Selenium to craft approximately 14-15 tests. The integration of Selenium with our CI/CD pipeline in Gitlab proved instrumental, enhancing our ability to identify and address new bugs before deployment. As the product's development progressed, the necessity arose to adapt the tests to accommodate new features, prompting the occasional creation of additional tests to augment our regression test suites.
While Selenium initially met our requirements, a significant challenge emerged as the tests exhibited flakiness, frequently reporting false positives and false negatives. This drawback became counterproductive, consuming more time than manual testing following system changes. Recognizing the need for a more reliable solution, our quest led us to Playwright.
Initial thoughts
Playwright received numerous positive feedback for its reliability, prompting me to explore its capabilities for our testing needs. Initially, I attempted to recreate our tests using Python, given its compatibility with Playwright. However, I encountered limitations as certain user-friendly features of Playwright were unavailable in the Python implementation. As a result, despite my limited proficiency in JavaScript and TypeScript, I opted to develop tests using TypeScript. The framework's ease of use became apparent, with any challenges arising primarily from my relative unfamiliarity with the programming language.
Key benefits
Installing Playwright is a straightforward process; it can be easily set up through npm without the need for additional installations, as opposed to Selenium.
Playwright boasts remarkable browser compatibility, eliminating concerns about code working seamlessly across various browsers like Safari and Chrome. In contrast to Selenium, where different drivers had to be created and installed for each browser, Playwright streamlines this process, requiring no extra packages for immediate functionality. This not only simplifies the code but also allows for the creation of separate browser contexts, facilitating parallel test execution.
The inclusion of Playwright's test runner UI enhances the testing experience. It simplifies test execution, debugging, code navigation, element inspection, console monitoring, and network analysis. Additionally, the UI provides an effortless way to view screenshots during test runs, all without the need for additional code. Configuration settings in the config file enable the framework to automatically save screenshots on failure and record execution videos.
Playwright's automatic waits offer a substantial advantage over Selenium, where extensive waits were necessary for proper functioning.
The reporting feature is integrated into Playwright, offering diverse output formats such as blob, xml, json, and html, as well as CLI options in both line and dot modes. Furthermore, users have the flexibility to implement custom reporting methods. This functionality proves especially beneficial when incorporating tests into CI/CD pipelines, particularly when there is a requirement to store reports within Jira.
Most notably, Playwright has proven to be highly reliable in our experiences so far. The singular drawback encountered pertains to the insufficient development of its documentation and a notably smaller community compared to Selenium.
Conclusions
Our transition from Selenium to Playwright at Tappointment has markedly enhanced our testing efficiency and reliability.
While Selenium initially met our requirements, the streamlined setup, unparalleled browser compatibility, user-friendly test runner UI, and integrated reporting features of Playwright have proven instrumental in overcoming challenges and optimizing our testing processes.
Despite encountering a limitation in Playwright's documentation and a smaller community, the benefits in terms of reliability and ease of use have decisively outweighed this singular drawback.
In our commitment to delivering high-quality software products, we persistently explore optimal solutions for our automated testing requirements, with Playwright currently serving as our preferred framework.
