Case Study Selenium

Customer story

Selenium’s builds with Bazel and the EngFlow platform are 10x faster and cover all major operating systems and browser versions.

faster builds & tests

10x

build time

From 5 hours down to 30 minutes

About Selenium

Selenium is a suite of open source tools for automating web browsers, primarily for testing web applications across different browsers and operating systems. At its core is a large polyglot repository, with over 200k lines of code written Java, C#, JavaScript, Rust, Ruby and Python, as well as hundreds of tests.

Remember before the equivalent test runs were taking up to 5 hours, and now they’re comfortably under 30 minutes. Our RBE build times are So. Much. Faster. It’s extraordinary.

Simon Stewart

Ex-project lead of Selenium

Long CI runs waste developer time

As Selenium grew in usefulness and popularity, more and more scenarios needed to be tested across multiple web browsers (Chrome, Firefox…), browser versions (dev, beta…) and operating systems. This caused CI times in GitHub to quickly spiral out of control – it took just a few months to go from 45 minutes to over 5 hours. This led to developers waiting incredibly long times before finding out if their Pull Requests had passed all the tests, or before discovering new bugs.

Bazel and EngFlow to the rescue

This CI runtime crisis motivated the team to expand its Bazel usage and adopt EngFlow’s Remote Build Execution platform. Doing both of these things empowered Selenium with clean, transparent, functional and hermetic builds, where build steps and testing steps could be interwoven thanks to Bazel’s efficient build graph, as well as heavily cached and parallelized across many machines thanks to EngFlow’s enterprise-grade remote cache and execution service.

We haven’t had to rewrite our tests, we haven’t done a huge amount of changes, but we’ve managed to slash our build time from 5 hours to under 30 minutes. And we could go faster, there are things we could do in order to squeeze a bit more performance, but we haven’t needed to do that.

Simon Stewart

Ex-project lead of Selenium

Initial surprises and quick wins

Switching the first tests to run on EngFlow was completely seamless: All it took was EngFlow providing a Remote Execution endpoint, and Selenium making a small configuration change for the first tests to be running remotely. The browser-based tests took a bit more effort, even with Bazel already managing browser and driver versions, because they required the ability to choose the Docker image the build actions were executed on – but with EngFlow’s help this was also straightened out.

When running the entire Chrome test suite on EngFlow for the first time, many tests were immediately green out-of-the-box and completed 5x faster, which was a good place to be starting from. Running all the linters, which could take over 1 hour in GitHub, took just 5 minutes on EngFlow – that’s a 12x speed-up!

{ Faster, Simpler } — Choose two

EngFlow doesn’t just make builds and tests faster, it also lets you simplify them.

You can see this for yourself in EngFlow’s Selenium Build and Test UI:

Left: GitHub Actions page with link to EngFlow Build and Test UI
Right: EngFlow Build and Test UI, showing a richer and more actionable view of a Bazel test failure

For failed tests, developers get a nicely rendered view of which test cases failed and why, as well as everything they need to dig in. For example, in the figure above, the terminal output on the left prints out that a target failed and includes the path to a log file containing more details. In EngFlow’s UI, the failure reason is shown inline, as well as a summary of which of the 20 test cases failed.

EngFlow also helps you simplify complex, expensive and error-prone platform-specific setups. Builds and tests specify how each individual action was configured and run, which can be analyzed to fix problems and improve their performance.

We recently migrated our .Net build to run on the EngFlow cluster too, and that’s simplified a pile of our testing. It’s allowed us to slim down our language and platform-specific builds (which were taking hours and often failed) to things that are far simpler.

Simon Stewart

Ex-project lead of Selenium

Run build and test as production infrastructure

EngFlow does more than make builds faster. It improves the operating performance of the whole engineering organization. Talk to a build expert about your codebase.