Scenario

Visual testing – find defects over thousands of images

SmartBear Software needed to decommission a product called CrossBrowserTesting, yet knew some of its technology, the DeviceCloud was still very valuable. The demand for QA automation, especially with regression tests, was growing. New features that leverage Artificial Intelligence and Machine Learning were highly anticipated.

Team

- VP of User Experience

- Product Manager

- Product Designer

- Tech Director

- 3 Developers


Methods

  • - Product /Market fit
  • - Competitive analysis
  • - User research & interviews
  • - Interface audit
  • - Prototyping
  • - Analytics & Product testing

Goals

  • - Rejuvenate key product
  • - Strategize feature list
  • - Streamline QA testing workflow
  • - Onboarding for all personas
  • - Branding & Design System

Contact

email@domain.com

000-000-000


— Instagram

— Twitter

— Facebook

The first move was to make sure we knew where are customers are, both current SmartBear users and new users

Remote co-workers worked on whiteboards to figure out how we might better organize projects, tests, and the results.

We did a Competitive Analysis to clarify the features that we knew would set SmartBear as a market leader

We kept access to SmartBear DeviceCloud knowing that capturing images from real devices, not emulators was great feature of our new product

Challenges

Moving fast to make a new product

Market and UX research came together with Developers to craft a strategy for the new product making sure the features would be make the best tool for visual testing

Product/Market Fit

- Collaborate with product line

- Define value for both current and new customers


Whiteboarding

  • - Crafting features list
  • - Gathering requirements
  • - Define resources and needs

Competitive Analysis

  • - Define two main competitors
  • - Refine and prioritize our feature list
  • - Evaluate information architecture

Features reviews

– Confirm best use case for current infrastructure

- Theorize the how the new features would plug-in

We fixed issues that analytics found in beta testing allowing non-technical personas to explore VisualTest for themselves with a Sample Project

Starting a project showed large icons that explained the how images would be compared

Developers were able get their tokens and integrate using a code snippet for their development language and frameworks, some tests begin to produce images in :30 seconds

The first test set baseline images and allowing for changes to be evaluated

Smartbear simplified complex structure of Projects, Coverage, Status, Test Results so that user could easily approve the automatic evaluation of 1,000s of images



Default comparisons are viewed in Side by Side, showing changes, additions, and subtractions

Adding comments to the images allows for more context on test results

Images included in a test can be managed easily

Results

Launched VisualTest in 6 months with new sales

Our team launched a new product called VisualTest. We went from a strategic idea to a real product, gaining it’s first paying customer in 6 months. VisualTest collaborated with other products in SmartBear's product line, like BitBar, giving current customers more testing options, easily accessed for a very affordable cost.


The UI design we crafted matched SmartBear’s existing product line and began adopting SmartBear’s new customized Design System.


Take aways

What we learned along the way

Onboarding is not afterthought. Having a consideration for how your different customers begin to interact with your new product should be designed in tandem with features and workflows.


Using your resources to move quickly. So many good decisions were thought out before development began because of the demanding go to market strategy, it forced us to set goals and quickly achieve them.