Growth and Change – MeasuringU

Who cares what happened 15 or 20 years ago? Though technology changes fast, some of the most important questions in UX research are enduring. Preparing for the future means understanding the past.

We’re celebrating our 20th anniversary at MeasuringU (2005–2025). For us, it’s less about popping the champagne and more about reflecting on how the UX industry has changed and how we have helped shape some of that change through measurement.

Some things have changed a lot while others haven’t. We looked back at key moments, reviewing influential publications and events to describe the story of how our company and the industry have evolved.

We’ve divided the MeasuringU timeline into three epochs. In each epoch, we briefly describe the industry trends, our company milestones, and the state of the art (including our contributions) to enduring UX topics of sample size estimation, online UX tools, usability testing, UX data analysis, and UX metrics.

In our previous article on the history of MeasuringU, we covered the foundational years from 1998 to 2008. This period included the founding of the company, the launching of the website, and the first of many collaborations and publications between Jeff Sauro and Jim Lewis, including the recommendation of adjusting binary confidence intervals for more accurate analysis.

As we continue to celebrate our 20^th anniversary, we move to our next major period, discussing our growth and change from 2009 through 2015.

Usability Becomes User Experience

The state of the economy at large was terrible in 2009. The world was working to recover from the Great Recession. Growth was slow and fragile.

The effect on UX jobs was additional cost cutting and very poor UX job numbers. One bright spot was expansion in the tech sector, much of that due to investments in cloud computing and mobile technology in the wake of the iPhone. Many UX designers adopted a “mobile first” philosophy to ensure a good mobile experience before scaling up to larger screens. The year 2010 saw the introduction of responsive web design, which automatically adapts layouts to different screen sizes, and the publication of Beyond the Usability Lab by Bill Albert, Tom Tullis, and Donna Tedesco. In 2012, the Usability Professionals Association (UPA) officially changed its name to the User Experience Professionals Association (UXPA).

Company Milestones

Events and Books

We hired our first employee during this period, and in 2012 we hosted the first Lean UX Denver conference (which morphed into our annual UX measurement bootcamp the next year).

Our data collection efforts really started taking off, and we began to publish our industry reports on the relationship between metrics like the SUS and NPS, which, due to their popularity, we continue to produce every few years.

We also started the internal process of creating our own standardized questionnaire to measure the quality of website experiences (the genesis of the SUPR-Q^®).

In 2010, Jeff published his first book, A Practical Guide to Measuring Usability, shortly followed in 2011 by A Practical Guide to the System Usability Scale. The following year marked the publication of the first edition of the highly regarded book Quantifying the User Experience and its Excel/R companion, both coauthored by Jeff and Jim (Figure 1).​

Figure 1: MeasuringU’s first four books.

Sample Size

The Most Comprehensive Guide to Sample Size Selection

The first edition of Quantifying the User Experience had two chapters explaining how to conduct sample size estimation for summative studies (Chapter 6) and formative studies (Chapter 7)—the most comprehensive treatment of those topics in the UX literature.

UX Online Tools

Unmoderated Tools Extend to Mobile

Before this time period, most mobile usability testing was moderated in person using cameras to capture interactions (e.g., the MOD 1000, our brief foray into physical product development). Around 2011, online tools (e.g., UserZoom, Loop11, and Usabilla) began to present remote studies on mobile devices.

Usability Testing

How Reliable are Quantitative Measures of Usability?

MeasuringU participated in the eighth comparative usability measurement study (CUE-8) in 2009 (Figure 2). Fifteen experienced professional usability teams simultaneously and independently measured a baseline for the usability of the car rental website Budget.com. The teams did not get exactly the same results, but “the reported measurements from several teams—sometimes a majority—agreed quite well.”

Figure 2: Publication of the results of CUE-8 in the Journal of User Experience (formerly the Journal of Usability Studies).

Data Analysis

Measuring Efficiency

In Quantifying the User Experience, we had multiple chapters explaining how to analyze the fundamental types of UX metrics (binary-discrete, rating scales, and time) and fundamental types of analysis (estimation, comparison with a benchmark, and comparison of products). We provided comparison methods that were appropriate when all measures were collected on the same people (within-subjects) or when different people experienced each product (between-subjects). Acknowledging the reality that many UX studies have small samples, we scoured the statistical literature for methods that work well for both small and large samples.

We continued to publish about how to use adjusted-Wald binomial confidence intervals in UX research. At the HCII conference in 2009, Jeff published a method for simplifying keystroke level modeling (KLM) by combining simple into composite operators. At CHI 2010, Jeff and Jim presented findings about the best ways to compute average task times given its skewed distribution and demonstrated that for small samples (n n > 25, you can use either the median or the geometric mean).

UX Metrics

Is There Quantitative Evidence of Usability?

During this period, MeasuringU made numerous significant contributions to advancements in UX metrics.

Since the publication of the SUM method, there had been an open question about the legitimacy of combining different UX metrics into a single score. It’s easier to argue for combination if the metrics are correlated, but a 2007 meta-analysis by Kasper Hornbæk, a really good researcher who assembled this data, indicated they were not. But, like all meta-analyses, the results are only as good as the inputs. A careful reading of that meta-analysis indicated their dataset wasn’t representative of industrial usability testing because they included many academic studies that weren’t focused on effectiveness, efficiency, and satisfaction.

In response, we collected data from 90 industrial usability test reports to understand the relationships among prototypical usability metrics. We found the correlations were strong when measured at the task level, and using principal components and factor analyses, we provided evidence for an underlying construct of general usability with objective and subjective factors (Figure 3). The publication of this research in CHI 2008 was nominated for a best paper award.

Figure 3: Correlations among prototypical usability metrics provided evidence for a general construct of usability.

At the same time, we were experimenting with new ways of measuring the perceived ease of tasks. When you have a lot of tasks, every questionnaire item adds up, so we experimented with three single-item candidates. Jeff and Joe Dumas adapted an ease item that Tedesco and Tullis had evaluated in 2006 and found it performed as well as or better than two other measures (SMEQ and usability magnitude estimation). This was the genesis of our Single Ease Question (SEQ^®), and the publication of this research at CHI 2008 was also nominated for a best paper award.

In 2011, we had numerous publications about the SUS, including Jeff’s second book (A Practical Guide to the System Usability Scale), Jeff’s Sustisfied article in User Experience Magazine (discussing little-known facts about the SUS), and a CHI paper by Jeff and Jim (honorable mention for best paper) that, using the SUS as a model, demonstrated no significant measurement advantage for the standard practice of mixing positive and negative tone items relative to consistent use of a positive tone.

Industry Trends

UX Teams Continued to Grow

Recovery from the Great Recession continued, but slowly. In the U.S., unemployment fell from 7.9% (2013) to 5% (2015), but wage growth remained sluggish.

UX research trends included increased use of remote unmoderated usability testing (including mobile), increased adoption of standardized UX metrics (e.g., SUS, SEQ), and increased executive buy-in on the value of UX research and its effects on business metrics.

Company Milestones

Changing Our Name and Upping Our Game

Measuring Usability became MeasuringU. We have to give Jakob Nielsen credit for that. He and Jeff had a conversation at CHI 2012 discussing how to simplify our Twitter handle (@MsrUsability, sometimes misread as “Mrs. Usability” or even the French “Monsieur Usability”), settling on @MeasuringU. We liked the new handle so much we changed the company name and got a brand-new logo and mission.

We had our first UX measurement bootcamp in 2013. In 2015, Jeff published Customer Analytics For Dummies to demonstrate the fusion between customer experience (CX) and UX in the application of quantitative research activities, an especially valuable resource for researchers who work with both departments (Figure 4).

Figure 4: Cover of Jeff’s book, Customer Analytics For Dummies.

Sample Size

Simplifying Sample Size Guidance

On the MeasuringU blog, Jeff tackled sample size topics like “How to Find the Sample Size for 8 Common Research Designs.” This was one of the first MeasuringU blog articles featuring tables to simplify sample size estimation for common UX research goals.

UX Online Tools

The Birth of MUiQ

With our increasing use of tools for conducting unmoderated UX research, in 2014 we developed the first version of our MeasuringU Intelligent Questioning software (MUiQ^®). This version could take data from other online tools for enhanced analysis using the methods we’d published in Quantifying the User Experience.

Usability Testing

More Severe Problems Aren’t Necessarily Uncovered First in a Usability Test

In the early 1990s, a controversy arose regarding whether problem severity affected the frequency of problem discovery. In 2014, Jeff published an analysis of data from nine formative usability studies and found that only one had a significant correlation between problem severity and frequency; the mean correlation across the studies was just .06. This demonstrated that, consistent with the math of the cumulative binomial probability formula, the only strong driver of problem discovery is sample size—more severe problems are not discovered more quickly than less severe problems. As Jeff recalls, “We had enough data to answer one of the enduring questions about formative usability studies. It only took 25 years from the original research but illustrates one of the motivations for the establishment of MeasuringU—to create a company dedicated to being able to answer these types of questions in external publication rather than hiding them away in corporate R&D labs.”

Data Analysis

Extended Analytical Guidance to Card Sorting, Icon Testing, and Reliability

On the MeasuringU blog, Jeff wrote data analysis articles on card sorting, icon testing, and computing different types of reliability.

UX Metrics

SUPR-Q

In 2015, Jeff published the research that drove the development of the Standardized User Experience Percentile Rank Questionnaire (SUPR-Q^®). The research was conducted over five years involving 4,000 user responses to experiences with over 100 websites, resulting in the generation of an eight-item questionnaire that assesses website quality with four factors: usability, trust, appearance, and loyalty.

Table 1 summarizes the key topics for the MeasuringU timeline from its genesis through 2015.

Topics	1998–2004	2005–2008	2009–2012	2013–2015
Industry Trends	Dot-com crash	iPhone, social media	Great recession	Sluggish recovery
Company Milestones	Identify need for MU	Measuring Usability, LLC	Hired first employee	MeasuringU
Sample Size	“5 is enough” controversy	Some resolution of “5 is enough”	Quantifying the User Experience (Ch 6-7)	For eight common research designs
UX Online Tools	Mostly meeting software plus WebEffective	RelevantView, UserZoom	Mobile test capabilities	First version of MUiQ (analytics)
Usability Testing	Summative defined, formative in crisis	NIST project, CUE studies	Practical Guide	Discovery frequency and problem severity
Data Analysis	Need for common sense UX statistics	Adjusted-Wald binomial confidence intervals	Quantifying the User Experience (Ch 3-5)	Card sorting, icon testing, reliability
UX Metrics	ISO 9241-12, ANSI CIF, Six Sigma, NPS	SUM	Construct of usability, SEQ, SUS	SUPR-Q

Table 1: Summary timeline of key topics.

What’s to Come

We have one more article on the history of MeasuringU to cover the events from 2016 through 2025 (e.g., economic expansion–pandemic–recovery, more books, sample size tables, maturation of MUiQ, AI, PURE, NPS, SUPR-Qm, UX-Lite, TAC-10).