user research methods

• Cosima Mielke
• Mar 27, 2024

The Future Of User Research: Expert Insights And Key Trends

• Share on Twitter ,  LinkedIn

About The Author

Cosima has been an editor at SmashingMag since 2013. Whenever she’s not writing articles for the weekly Smashing Newsletter , she’s probably working on a … More about Cosima ↬

Email Newsletter

Weekly tips on front-end & UX . Trusted by 200,000+ folks.

This article has been kindly supported by our dear friends at Maze , the user research platform that empowers any company to build the right products faster by making user insights available at the speed of product development. Thank you!

How do product teams conduct user research today? How do they leverage user insights to make confident decisions and drive business growth? And what role does AI play? To learn more about the current state of user research and uncover the trends that will shape the user research landscape in 2024 and beyond , Maze surveyed over 1,200 product professionals between December 2023 and January 2024.

The Future of User Research Report summarized the data into three key trends that provide precious insights into an industry undergoing significant changes . Let’s take a closer look at the main findings from the report.

Trend 1: The Demand For User Research Is Growing

62% of respondents who took the Future of User Research survey said the demand for user research has increased in the past 12 months. Industry trends like continuous product discovery and research democratization could be contributing to this growth, along with recent layoffs and reorganizations in the tech industry.

Emma Craig, Head of UX Research at Miro, sees one reason for this increase in the uncertain times we’re living in. Under pressure to beat the competition, she sensed a “shift towards more risk-averse attitudes, where organizations feel they need to ‘get it right’ the first time.” By conducting user research, organizations can mitigate risk and clarify the strategy of their business or product.

Research Is About Learning

As the Future of User Research report found out, organizations are leveraging research to make decisions across the entire product development lifecycle . The main consumers of research are design (86%) and product (83%) teams, but it’s also marketing, executive teams, engineering, data, customer support, and sales who rely on the results from user research to inform their decision-making.

As Roberta Dombrowski, Research Partner at Maze, points out:

“At its core, research is about learning. We learn to ensure that we’re building products and services that meet the needs of our customers. The more we invest in growing our research practices and team, the higher our likelihood of meeting these needs.”

Benefits And Challenges Of Conducting User Research

As it turns out, the effort of conducting user research on a regular basis pays off. 85% of respondents said that user research improved their product’s usability , 58% saw an increase in customer satisfaction , and 44% in customer engagement .

Connecting research insights to business outcomes remains a key challenge, though. While awareness for measuring research impact is growing (73% of respondents track the impact of their research), 41% reported they find it challenging to translate research insights into measurable business outcomes . Other significant challenges teams face are time and bandwidth constraints (62%) and recruiting the right participants (60%).

Growing A Research Mindset

With the demand for user research growing, product teams need to find ways to expand their research initiatives. 75% of the respondents in the Maze survey are planning to scale research in the next year by increasing the number of research studies, leveraging AI tools, and providing training to promote research democratization.

Janelle Ward, Founder of Janelle Ward Insights, sees great potential in growing research practices, as an organization will grow a research mindset in tandem. She shares:

“Not only will external benefits like competitive advantage come into play, but employees inside the organization will also better understand how and why important business decisions are made, resulting in more transparency from leadership and a happier and more thriving work culture for everyone.”

Trend 2: Research Democratization Empowers Stronger Decision-Making

Research democratization involves empowering different teams to run research and get access to the insights they need to make confident decisions. The Future of User Research Report shows that in addition to researchers, product designers (61%), product managers (38%), and marketers (17%) conduct user research at their companies to inform their decision-making.

Teams with a democratized research culture reported a greater impact on decision-making. They are 2× more likely to report that user research influences strategic decisions , 1.8× more likely to state that it impacts product decisions, and 1.5× more likely to express that it inspires new product opportunities.

The User Researcher’s New Role

Now, if more people are conducting user research in an organization, does this mark the end of the user researcher role? Not at all. Scaling research through democratization doesn’t mean anyone can do any type of research. You’ll need the proper checks and balances to allow everyone to participate in research responsibly and effectively. The role is shifting from a purely technical to an educational role where user researchers become responsible for guiding the organization in its learning and curiosity.

To guarantee data quality and accuracy, user researchers can train partners on research methods and best practices and give them hands-on experience before they start their own research projects. This can involve having them shadow a researcher during a project, holding mock interviews, or leading collaborative analysis workshops.

Democratizing user research also means that UX researchers can open up time to focus on more complex research initiatives . While tactical research, such as usability testing, can be delegated to designers and product managers, UX researchers can conduct foundational studies to inform the product and business strategy.

User Research Tools And Techniques

It’s also interesting to see which tools and techniques product teams use to gather user insights. Maze (46%), Hotjar (26%), and UserTesting (24%) are the most widely used user research tools. When it comes to user research methods, product teams mostly turn to user interviews (89%), usability testing (85%), surveys (82%), and concept testing (56%).

According to Morgan Mullen, Lead UX Researcher at User Interviews, a factor to consider is the type of projects teams conduct. Most teams don’t change their information architecture regularly, which requires tree testing or card sorting. But they’re likely launching new features often, making usability testing a more popular research method.

Trend 3: New Technology Allows Product Teams To Significantly Scale Research

AI is reshaping how we work in countless ways, and user research is no exception. According to the Future of User Research Report, 44% of product teams are already using AI tools to run research and an additional 41% say they would like to adopt AI tools in the future.

ChatGPT is the most widely-used AI tool for conducting research (82%), followed by Miro AI (20%), Notion AI (18%), and Gemini (15%). The most commonly used research tools with AI features are Maze AI (15%), UserTesting AI (9%), and Hotjar AI (5%).

The Strengths Of AI

The tactical aspect of research is where AI truly shines. More than 60% of respondents use AI to analyze user research data , 54% for transcription , 48% for generating research questions, and 45% for synthesis and reporting . By outsourcing these tasks to artificial intelligence, respondents reported that their team efficiency improved (56%) and turnaround time for research projects decreased (50%) — freeing up more time to focus on the human and strategic side of research (35%).

The Irreplaceable Value Of Research

While AI is great at tackling time-consuming, tactical tasks, it is not a replacement for a skilled researcher. As Kate Pazoles, Head of Flex User Research at Twilio, points out, we can think of AI as an assistant. The value lies in connecting the dots and uncovering insights with a level of nuance that only UX researchers possess.

Jonathan Widawski, co-founder and CEO at Maze, sums up the growing role that AI plays in user research as follows:

“AI will be able to support the entire research process, from data collection to analysis. With automation powering most of the tactical aspects, a company’s ability to build products fast is no longer a differentiating factor. The key now lies in a company’s ability to build the right product — and research is the power behind all of this.”

Looking Ahead

With teams adopting a democratized user research culture and AI tools on the rise, the user researcher’s role is shifting towards that of a strategic partner for the organization .

Instead of gatekeeping their knowledge, user researchers can become facilitators and educate different teams on how to engage with customers and use those insights to make better decisions. By doing so, they help ensure research quality and accuracy conducted by non-researchers, while opening up time to focus on more complex, strategic research . Adopting a research mindset also helps teams value user research more and foster a happier, thriving work culture . A win-win for the organization, its employees, and customers.

If you’d like more data and insights, read the full Future of User Research Report by Maze here .

Integrations

What's new?

Prototype Testing

Live Website Testing

Feedback Surveys

Interview Studies

Card Sorting

Tree Testing

In-Product Prompts

Participant Management

Automated Reports

Templates Gallery

Choose from our library of pre-built mazes to copy, customize, and share with your own users

Browse all templates

Financial Services

Tech & Software

Product Designers

Product Managers

User Researchers

By use case

Concept & Idea Validation

Wireframe & Usability Test

Content & Copy Testing

Feedback & Satisfaction

Content Hub

Educational resources for product, research and design teams

Explore all resources

Question Bank

Research Maturity Model

Guides & Reports

Help Center

Future of User Research Report

The Optimal Path Podcast

User Research

Mar 21, 2024

Creating a research hypothesis: How to formulate and test UX expectations

A research hypothesis helps guide your UX research with focused predictions you can test and learn from. Here’s how to formulate your own hypotheses.

Armin Tanovic

All great products were once just thoughts—the spark of an idea waiting to be turned into something tangible.

A research hypothesis in UX is very similar. It’s the starting point for your user research; the jumping off point for your product development initiatives.

Formulating a UX research hypothesis helps guide your UX research project in the right direction, collect insights, and evaluate not only whether an idea is worth pursuing, but how to go after it.

In this article, we’ll cover what a research hypothesis is, how it's relevant to UX research, and the best formula to create your own hypothesis and put it to the test.

Test your hypothesis with Maze

Maze lets you validate your design and test research hypotheses to move forward with authentic user insights.

What defines a research hypothesis?

A research hypothesis is a statement or prediction that needs testing to be proven or disproven.

Let’s say you’ve got an inkling that making a change to a feature icon will increase the number of users that engage with it—with some minor adjustments, this theory becomes a research hypothesis: “ Adjusting Feature X’s icon will increase daily average users by 20% ”.

A research hypothesis is the starting point that guides user research . It takes your thought and turns it into something you can quantify and evaluate. In this case, you could conduct usability tests and user surveys, and run A/B tests to see if you’re right—or, just as importantly, wrong .

A good research hypothesis has three main features:

• Specificity: A hypothesis should clearly define what variables you’re studying and what you expect an outcome to be, without ambiguity in its wording
• Relevance: A research hypothesis should have significance for your research project by addressing a potential opportunity for improvement
• Testability: Your research hypothesis must be able to be tested in some way such as empirical observation or data collection

What is the difference between a research hypothesis and a research question?

Research questions and research hypotheses are often treated as one and the same, but they’re not quite identical.

A research hypothesis acts as a prediction or educated guess of outcomes , while a research question poses a query on the subject you’re investigating. Put simply, a research hypothesis is a statement, whereas a research question is (you guessed it) a question.

For example, here’s a research hypothesis: “ Implementing a navigation bar on our dashboard will improve customer satisfaction scores by 10%. ”

This statement acts as a testable prediction. It doesn’t pose a question, it’s a prediction. Here’s what the same hypothesis would look like as a research question: “ Will integrating a navigation bar on our dashboard improve customer satisfaction scores? ”

The distinction is minor, and both are focused on uncovering the truth behind the topic, but they’re not quite the same.

Why do you use a research hypothesis in UX?

Research hypotheses in UX are used to establish the direction of a particular study, research project, or test. Formulating a hypothesis and testing it ensures the UX research you conduct is methodical, focused, and actionable. It aids every phase of your research process , acting as a north star that guides your efforts toward successful product development .

Typically, UX researchers will formulate a testable hypothesis to help them fulfill a broader objective, such as improving customer experience or product usability. They’ll then conduct user research to gain insights into their prediction and confirm or reject the hypothesis.

A proven or disproven hypothesis will tell if your prediction is right, and whether you should move forward with your proposed design—or if it's back to the drawing board.

Formulating a hypothesis can be helpful in anything from prototype testing to idea validation, and design iteration. Put simply, it’s one of the first steps in conducting user research.

Whether you’re in the initial stages of product discovery for a new product, a single feature, or conducting ongoing research, a strong hypothesis presents a clear purpose and angle for your research It also helps understand which user research methodology to use to get your answers.

What are the types of research hypotheses?

Not all hypotheses are built the same—there are different types with different objectives. Understanding the different types enables you to formulate a research hypothesis that outlines the angle you need to take to prove or disprove your predictions.

Here are some of the different types of hypotheses to keep in mind.

Null and alternative hypotheses

While a normal research hypothesis predicts that a specific outcome will occur based upon a certain change of variables, a null hypothesis predicts that no difference will occur when you introduce a new condition.

By that reasoning, a null hypothesis would be:

• Adding a new CTA button to the top of our homepage will make no difference in conversions

Null hypotheses are useful because they help outline what your test or research study is trying to dis prove, rather than prove, through a research hypothesis.

An alternative hypothesis states the exact opposite of a null hypothesis. It proposes that a certain change will occur when you introduce a new condition or variable. For example:

• Adding a CTA button to the top of our homepage will cause a difference in conversion rates

Simple hypotheses and complex hypotheses

A simple hypothesis is a prediction that includes only two variables in a cause-and-effect sequence, with one variable dependent on the other. It predicts that you'll achieve a particular outcome based on a certain condition. The outcome is known as the dependent variable and the change causing it is the independent variable .

For example, this is a simple hypothesis:

• Including the search function on our mobile app will increase user retention

The expected outcome of increasing user retention is based on the condition of including a new search function. But, what happens when there are more than two factors at play?

We get what’s called a complex hypothesis. Instead of a simple condition and outcome, complex hypotheses include multiple results. This makes them a perfect research hypothesis type for framing complex studies or tracking multiple KPIs based on a single action.

Building upon our previous example, a complex research hypothesis could be:

• Including the search function on our mobile app will increase user retention and boost conversions

Directional and non-directional hypotheses

Research hypotheses can also differ in the specificity of outcomes. Put simply, any hypothesis that has a specific outcome or direction based on the relationship of its variables is a directional hypothesis . That means that our previous example of a simple hypothesis is also a directional hypothesis.

Non-directional hypotheses don’t specify the outcome or difference the variables will see. They just state that a difference exists. Following our example above, here’s what a non-directional hypothesis would look like:

• Including the search function on our mobile app will make a difference in user retention

In this non-directional hypothesis, the direction of difference (increase/decrease) hasn’t been specified, we’ve just noted that there will be a difference.

The type of hypothesis you write helps guide your research—let’s get into it.

How to write and test your UX research hypothesis

Now we’ve covered the types of research hypothesis examples, it’s time to get practical.

Creating your research hypothesis is the first step in conducting successful user research.

Here are the four steps for writing and testing a UX research hypothesis to help you make informed, data-backed decisions for product design and development.

1. Formulate your hypothesis

Start by writing out your hypothesis in a way that’s specific and relevant to a distinct aspect of your user or product experience. Meaning: your prediction should include a design choice followed by the outcome you’d expect—this is what you’re looking to validate or reject.

Your proposed research hypothesis should also be testable through user research data analysis. There’s little point in a hypothesis you can’t test!

Let’s say your focus is your product’s user interface—and how you can improve it to better meet customer needs. A research hypothesis in this instance might be:

• Adding a settings tab to the navigation bar will improve usability

By writing out a research hypothesis in this way, you’re able to conduct relevant user research to prove or disprove your hypothesis. You can then use the results of your research—and the validation or rejection of your hypothesis—to decide whether or not you need to make changes to your product’s interface.

2. Identify variables and choose your research method

Once you’ve got your hypothesis, you need to map out how exactly you’ll test it. Consider what variables relate to your hypothesis. In our case, the main variable of our outcome is adding a settings tab to the navigation bar.

Once you’ve defined the relevant variables, you’re in a better position to decide on the best UX research method for the job. If you’re after metrics that signal improvement, you’ll want to select a method yielding quantifiable results—like usability testing . If your outcome is geared toward what users feel, then research methods for qualitative user insights, like user interviews , are the way to go.

3. Carry out your study

It’s go time. Now you’ve got your hypothesis, identified the relevant variables, and outlined your method for testing them, you’re ready to run your study. This step involves recruiting participants for your study and reaching out to them through relevant channels like email, live website testing , or social media.

Given our hypothesis, our best bet is to conduct A/B and usability tests with a prototype that includes the additional UI elements, then compare the usability metrics to see whether users find navigation easier with or without the settings button.

We can also follow up with UX surveys to get qualitative insights and ask users how they found the task, what they preferred about each design, and to see what additional customer insights we uncover.

💡 Want more insights from your usability tests? Maze Clips enables you to gather real-time recordings and reactions of users participating in usability tests .

4. Analyze your results and compare them to your hypothesis

By this point, you’ve neatly outlined a hypothesis, chosen a research method, and carried out your study. It’s now time to analyze your findings and evaluate whether they support or reject your hypothesis.

Look at the data you’ve collected and what it means. Given that we conducted usability testing, we’ll want to look to some key usability metrics for an indication of whether the additional settings button improves usability.

For example, with the usability task of ‘ In account settings, find your profile and change your username ’, we can conduct task analysis to compare the times spent on task and misclick rates of the new design, with those same metrics from the old design.

If you also conduct follow-up surveys or interviews, you can ask users directly about their experience and analyze their answers to gather additional qualitative data . Maze AI can handle the analysis automatically, but you can also manually read through responses to get an idea of what users think about the change.

By comparing the findings to your research hypothesis, you can identify whether your research accepts or rejects your hypothesis. If the majority of users struggle with finding the settings page within usability tests, but had a higher success rate with your new prototype, you’ve proved the hypothesis.

However, it's also crucial to acknowledge if the findings refute your hypothesis rather than prove it as true. Ruling something out is just as valuable as confirming a suspicion.

In either case, make sure to draw conclusions based on the relationship between the variables and store findings in your UX research repository . You can conduct deeper analysis with techniques like thematic analysis or affinity mapping .

UX research hypotheses: four best practices to guide your research

Knowing the big steps for formulating and testing a research hypothesis ensures that your next UX research project gives you focused, impactful results and insights. But, that’s only the tip of the research hypothesis iceberg. There are some best practices you’ll want to consider when using a hypothesis to test your UX design ideas.

Here are four research hypothesis best practices to help guide testing and make your UX research systematic and actionable.

Align your hypothesis to broader business and UX goals

Before you begin to formulate your hypothesis, be sure to pause and think about how it connects to broader goals in your UX strategy . This ensures that your efforts and predictions align with your overarching design and development goals.

For example, implementing a brand new navigation menu for current account holders might work for usability, but if the wider team is focused on boosting conversion rates for first-time site viewers, there might be a different research project to prioritize.

Create clear and actionable reports for stakeholders

Once you’ve conducted your testing and proved or disproved your hypothesis, UX reporting and analysis is the next step. You’ll need to present your findings to stakeholders in a way that's clear, concise, and actionable. If your hypothesis insights come in the form of metrics and statistics, then quantitative data visualization tools and reports will help stakeholders understand the significance of your study, while setting the stage for design changes and solutions.

If you went with a research method like user interviews, a narrative UX research report including key themes and findings, proposed solutions, and your original hypothesis will help inform your stakeholders on the best course of action.

Consider different user segments

While getting enough responses is crucial for proving or disproving your hypothesis, you’ll want to consider which users will give you the highest quality and most relevant responses. Remember to consider user personas —e.g. If you’re only introducing a change for premium users, exclude testing with users who are on a free trial of your product.

You can recruit and target specific user demographics with the Maze Panel —which enables you to search for and filter participants that meet your requirements. Doing so allows you to better understand how different users will respond to your hypothesis testing. It also helps you uncover specific needs or issues different users may have.

Involve stakeholders from the start

Before testing or even formulating a research hypothesis by yourself, ensure all your stakeholders are on board. Informing everyone of your plan to formulate and test your hypothesis does three things:

Firstly, it keeps your team in the loop . They’ll be able to inform you of any relevant insights, special considerations, or existing data they already have about your particular design change idea, or KPIs to consider that would benefit the wider team.

Secondly, informing stakeholders ensures seamless collaboration across multiple departments . Together, you’ll be able to fit your testing results into your overall CX strategy , ensuring alignment with business goals and broader objectives.

Finally, getting everyone involved enables them to contribute potential hypotheses to test . You’re not the only one with ideas about what changes could positively impact the user experience, and keeping everyone in the loop brings fresh ideas and perspectives to the table.

Test your UX research hypotheses with Maze

Formulating and testing out a research hypothesis is a great way to define the scope of your UX research project clearly. It helps keep research on track by providing a single statement to come back to and anchor your research in.

Whether you run usability tests or user interviews to assess your hypothesis—Maze's suite of advanced research methods enables you to get the in-depth user and customer insights you need.

Frequently asked questions about research hypothesis

What is the difference between a hypothesis and a problem statement in UX?

A research hypothesis describes the prediction or method of solving that problem. A problem statement, on the other hand, identifies a specific issue in your design that you intend to solve. A problem statement will typically include a user persona, an issue they have, and a desired outcome they need.

How many hypotheses should a UX research problem have?

Technically, there are no limits to the amount of hypotheses you can have for a certain problem or study. However, you should limit it to one hypothesis per specific issue in UX research. This ensures that you can conduct focused testing and reach clear, actionable results.

This paper is in the following e-collection/theme issue:

Published on 29.3.2024 in Vol 26 (2024)

Usability of Health Care Price Transparency Data in the United States: Mixed Methods Study

Authors of this article:

Original Paper

• Negar Maleki 1 , PhD   ; 
• Balaji Padmanabhan 2 , PhD   ; 
• Kaushik Dutta 1 , PhD  

1 School of Information Systems and Management, Muma College of Business, University of South Florida, Tampa, FL, United States

2 Decision, Operations & Information Technologies Department, Robert H. Smith School of Business, University of Maryland, College Park, MD, United States

Corresponding Author:

Negar Maleki, PhD

School of Information Systems and Management

Muma College of Business

University of South Florida

4202 E Fowler Avenue

Tampa, FL, 33620

United States

Phone: 1 8139742011

Email: [email protected]

Background: Increasing health care expenditure in the United States has put policy makers under enormous pressure to find ways to curtail costs. Starting January 1, 2021, hospitals operating in the United States were mandated to publish transparent, accessible pricing information online about the items and services in a consumer-friendly format within comprehensive machine-readable files on their websites.

Objective: The aims of this study are to analyze the available files on hospitals’ websites, answering the question—is price transparency (PT) information as provided usable for patients or for machines?—and to provide a solution.

Methods: We analyzed 39 main hospitals in Florida that have published machine-readable files on their website, including commercial carriers. We created an Excel (Microsoft) file that included those 39 hospitals along with the 4 most popular services—Current Procedural Terminology (CPT) 45380, 29827, and 70553 and Diagnosis-Related Group (DRG) 807—for the 4 most popular commercial carriers (Health Maintenance Organization [HMO] or Preferred Provider Organization [PPO] plans)—Aetna, Florida Blue, Cigna, and UnitedHealthcare. We conducted an A/B test using 67 MTurkers (randomly selected from US residents), investigating the level of awareness about PT legislation and the usability of available files. We also suggested format standardization, such as master field names using schema integration, to make machine-readable files consistent and usable for machines.

Results: The poor usability and inconsistent formats of the current PT information yielded no evidence of its usefulness for patients or its quality for machines. This indicates that the information does not meet the requirements for being consumer-friendly or machine readable as mandated by legislation. Based on the responses to the first part of the experiment (PT awareness), it was evident that participants need to be made aware of the PT legislation. However, they believe it is important to know the service price before receiving it. Based on the responses to the second part of the experiment (human usability of PT information), the average number of correct responses was not equal between the 2 groups, that is, the treatment group (mean 1.23, SD 1.30) found more correct answers than the control group (mean 2.76, SD 0.58; t 65 =6.46; P <.001; d =1.52).

Conclusions: Consistent machine-readable files across all health systems facilitate the development of tools for estimating customer out-of-pocket costs, aligning with the PT rule’s main objective—providing patients with valuable information and reducing health care expenditures.

Introduction

From 1970 to 2020, on a per capita basis, health care expenditures in the United States have increased sharply from US $353 per person to US $12,531 per person. In constant 2020 dollars, the increase was from US $1875 in 1970 to US $12,531 in 2020 [ 1 ]. The significant rise in health care expenses has put policy makers under enormous pressure to find ways to contain these expenditures. Price transparency (PT) in health care is 1 generally proposed strategy for addressing these problems [ 2 ] and has been debated for years [ 3 ]. Some economists believe that PT in health care will cut health care prices in the same way it has in other industries, while others argue that owing to the specific characteristics of the health care market, PT would not ameliorate rising health care costs. Price elasticity also does not typically apply in health care, since, if a problem gets severe, people will typically seek treatment regardless of cost, with the drawback that individuals learn of their health care costs after receiving treatment [ 4 ]. Complex billing processes, hidden insurer-provider contracts, the sheer quantity of third-party payers, and substantial quality differences in health care delivery are other unique aspects of health care that complicate the situation considerably.

The Centers for Medicare & Medicaid Services (CMS) mandated hospitals to post negotiated rates, including payer-specific negotiated costs, for 300 “shoppable services” beginning in January 2021. The list must include 70 CMS-specified services and an additional 230 services each hospital considers relevant to its patient population. Hospitals must include each third-party payer and their payer-specific fee when negotiating multiple rates for the same care. The data must be displayed simply, easily accessible (without requiring personal information from the patient), and saved in a machine-readable manner [ 5 ]. These efforts aim to facilitate informed patient decision-making, reduce out-of-pocket spending, and decrease health care expenditures. Former Secretary of Health and Human Services, Alex Azar, expressed a vision of hospital PT when declaring the new legislation “a patient-centered system that puts you in control and provides the affordability you need, the options and control you want, and the quality you deserve. Providing patients with clear, accessible information about the price of their care is a vital piece of delivering on that vision” [ 6 ].

Despite the legislation, it is not clear if people are actually engaging in using PT tools. For example, in 2007, New Hampshire’s HealthCost website was established, providing the negotiated price and out-of-pocket costs for 42 commonly used services by asking whether the patient is insured or their insurer and the zip code to post out-of-pocket costs in descending order. Mehrotra et al [ 7 ] examined this website over 3 years to understand how often and why these tools have mainly been used. Their analysis suggested that despite the growing interest in PT, approximately 1% of the state’s population used this tool. Low PT tool usage was also seen in other studies [ 8 - 10 ], suggesting that 3% to 12% of individuals who were offered the tool used it during the study period, and in all studies, the duration was at least 12 months. Thus, offering PT tools does not in itself lead to decreased total spending, since few people who have access to them use them to browse for lower-cost services [ 7 , 11 ].

In a recent paper, researchers addressed 1 possible reason for low engagement—lack of awareness. They implemented an extensive targeted online advertising campaign using Google Advertisements to increase awareness and assessed whether it increased New Hampshire’s PT website use. Their findings suggested that although lack of awareness is a possible reason for the low impact of PT tools in health care spending, structural factors might affect the use of health care information [ 12 ]. Individuals may not be able to exactly determine their out-of-pocket expenses from the information provided.

Surprisingly, there is little research on the awareness and usability of PT information after the current PT legislation went into effect. A recent study [ 13 ] highlighted the nonusability of existing machine-readable files for employers, policy makers, researchers, or consumers, and this paper adds to this literature by answering the question—is PT information as provided usable for patients or machines? Clearly, if it is of value to patients, it can be useful; the reason to take the perspective of machines was to examine whether this information as provided might also be useful for third-party programs that can extract information from the provided data (to subsequently help patients through other ways of presenting this information perhaps). We address this question through a combination of user experiments and data schema analysis. While there are recent papers that have also argued that PT data have deficiencies [ 13 , 14 ], ours is the first to combine user experiments with analysis of data schema from several hospitals in Florida to make a combined claim on value for patients and machines. We hope this can add to the discourse on PT and what needs to be done to extract value for patients and the health care system as a whole.

Impact of PT Tools

The impact of PT tools on consumers and health care facilities has been investigated in the literature. Some studies showed that consumers with access to PT tools are more likely to reduce forgone needed services over time. Moreover, consumers who use tools tend to find the lowest service prices [ 8 , 15 - 17 ]. A few studies investigated the impact of PT tools on the selection of health care facilities. They illustrated that some consumers tend to change health care facilities pursuing lower prices, while some others prefer to stay with expensive ones, although they are aware of some other facilities that offer lower prices [ 9 , 18 ]. Finally, some research studied the impact of PT tools on cost and showed that some consumers experienced no effect, while others experienced decreases in average consumer expenses [ 8 , 17 , 18 ]. However, the impact of PT tools on health care facilities is inconclusive, meaning different studies concluded different effects. Some stated that PT tools decrease the prices of imaging and laboratory services, while others said that although public charge disclosure lowers health care facility charges, the final prices remained unchanged [ 17 , 18 ].

Legislation Related Works

In a study, researchers considered 20 leading US hospitals to assess provided chargemasters to understand to what extent patients can obtain information from websites to determine the out-of-pocket costs [ 19 ]. Their findings showed that although all hospitals provided chargemasters on their websites, they rarely offered transparent information, making it hard for patients to determine out-of-pocket costs. Their analysis used advanced diagnostic imaging services to assess hospitals’ chargemasters since these are the most common services people look for. Mehrotra et al [ 7 ] also mentioned that the most common searches belonged to outpatient visits, magnetic resonance imaging (MRI), and emergency department visits. To this end, we used “MRI scan of the brain before and after contrast” as one of the shoppable services in our analysis. Another study examined imaging services in children’s hospitals (n=89), restricting the analysis to hospitals (n=35) that met PT requirements—published chargemaster rates, discounted cash prices, and payer-negotiated prices in a machine-readable file, and published costs for 300 common shoppable medical services in a consumer-friendly format. Their study revealed that, in addition to a broad range of imaging service charges, most hospitals lack the machine-readable file requirement [ 20 ].

Arvisais-Anhalt et al [ 21 ] identified 11 hospitals with available chargemasters in Dallas County to compare the prices of a wide range of available services. They observed significant variations for a laboratory test: partial thromboplastin time, a medication: 5 mg tablet of amlodipine, and a procedure: circumcision. Reddy et al [ 22 ] focus on New York State to assess the accessibility and usability of hospitals’ chargemasters from patients’ viewpoint. They found that 189 out of 202 hospitals had a locatable chargemaster on their home page. However, only 37 hospitals contain the Current Procedural Terminology (CPT) code, which makes those without the CPT code unusable due to the existence of many different descriptions for the same procedure; for example, an elective heart procedure had 34 entries. We add to this considerable literature by examining a subset of Florida hospitals.

In a competitive market, higher-quality goods and services require higher prices [ 23 ]. Based on this, Patel et al [ 24 ] examined the relationship between the Diagnosis-Related Group (DRG) chargemaster and quality measures. Although prior research found no convincing evidence that hospitals with greater costs also delivered better care [ 25 ], they discovered 2 important quality indicators that were linked to standard charges positively and substantially—mortality rate and readmission rates—which both are quality characteristics that are in line with economic theory. Moreover, Patel et al [ 24 ] studied the variety of one of the most commonly performed services (vaginal delivery) as a DRG code, which motivated us to select “Vaginal delivery without sterilization or D&C without CC/MCC” as another shoppable service in our analysis.

Ethical Considerations

All data used in this study, including the secondary data set obtained from hospitals’ websites and the data collected during the user experiment, underwent a thorough anonymization process. The study was conducted under protocols approved by the University of South Florida institutional review board (STUDY004145: “Effect of price transparency regulation (PTR) on the public decisions”) under HRP-502b(7) Social Behavioral Survey Consent. This approval encompassed the use of publicly available anonymized secondary data from hospitals’ websites, as well as a user experiment aimed at assessing awareness of the PT rule and the usability of hospitals’ files. No individual-specific data were collected during the experiment, which solely focused on capturing subjects’ awareness and opinions regarding the PT rule and associated files. At the onset of the experiment, participants were provided with a downloadable consent form and were allowed to withdraw their participation at any time. Survey participants were offered a US $2 reward, and their involvement was entirely anonymous.

Data Collection

According to CMS, “Starting January 1, 2021, each hospital operating in the United States will be required to provide clear, accessible pricing information online about the items and services they provide in two ways: 1- As a comprehensive machine-readable file with all items and services. 2- In a display of shoppable services in a consumer-friendly format.” As stated, files available on hospitals’ websites should be consumer-friendly, so the question of whether these files are for users arises. On the other hand, as stated, files should be machine-readable, so again the question of whether these files are for machines arises. Below we try to answer both questions in detail, respectively.

Value for Users: User Experiments

When a public announcement is disseminated, its efficacy relies on ensuring widespread awareness and facilitating practical use during times of necessity. Previous research on PT announcements has highlighted the challenges faced by patients in accurately estimating out-of-pocket expenses. However, a fundamental inquiry arises—are individuals adequately informed about the availability of tools that enable them to estimate their out-of-pocket costs for desired services? To address this, we conducted a survey to assess public awareness of PT legislation. The survey encompassed a range of yes or no and multiple-choice questions aimed at gauging participants’ familiarity with the PT rule in health care and their entitlement to obtain cost information prior to receiving a service. Additionally, we inquired about participants’ knowledge of resources for accessing pricing information and whether they were aware of the PT rule. Furthermore, we incorporated follow-up questions to ensure that the survey responses were not provided arbitrarily, thereby securing reliable and meaningful outcomes.

Moreover, considering the previously established evidence of subpar usability associated with the currently available files, we propose streamlining the existing files and developing a user-friendly and comprehensive document for conducting an A/B test. This test aims to evaluate which file better facilitates participants in accurately estimating their out-of-pocket costs. In collaboration with Florida Blue experts during biweekly meetings throughout the entire process outlined in this paper, the authors determined the optimal design for the summary table. This design, which presents prices in a more user-friendly format, enhancing overall participant comprehension, was used during the A/B testing. Participants were randomly assigned to either access the hospitals’ files or a meticulously constructed summary table, manually created in Excel, prominently displaying cost information (Please note that all files, including the hospitals’ files and our Excel file, are made available in the same format [Excel] on a cloud-based platform to eliminate any disparities in accessing the files. This ensures equitable ease of finding, downloading, and opening files, as accessing the hospitals’ files typically requires significant effort.). The experiment entailed presenting 3 distinct health-related scenarios and instructing participants to locate the price for the requested service. Subsequently, participants were asked to provide the hospital name, service price, insurer name, and insurance plan. Additionally, we sought feedback on the perceived difficulty of finding the requested service and their priority for selecting hospitals [ 26 ], followed by Likert scale questions to assess participants’ evaluation of the provided file’s efficacy in facilitating price retrieval.

The experiments were conducted to investigate the following questions: (1) Are the individuals aware of the PT legislation? and (2) Is the information provided usable for patients? To evaluate the usability of files found on websites, we selected 2 prevalent services based on existing literature and 2 other services recommended as high-demand ones by Florida Blue experts, Table 1 . Furthermore, meticulous efforts were made to ensure that both the control and treatment groups encountered identical circumstances, thus allowing for a systematic examination of the disparities solely attributable to variations in data representation.

a DRG: Diagnosis-Related Group.

b D&C: dilation and curettage.

c CC/MCC: complication or comorbidity/major complication or comorbidity.

d CPT: Current Procedural Terminology.

e MRI: magnetic resonance imaging.

Participants

A total of 67 adults (30 female individuals; mean 41.43, SD 12.39 years) were recruited on the Amazon Mechanical Turk platform, with no specific selection criteria other than being located in the United States.

We focused on 75 main hospitals (ie, the main hospital refers to distinguish a hospital from smaller clinics or specialized medical centers within the same health system) in the state of Florida. When we searched their websites for PT files (machine-readable files), only 89% (67/75) of hospitals included machine-readable files. According to the PT legislation, these files were supposed to contain information about 300 shoppable services. However, only 58% (39/67) of hospitals included information such as insurer prices in their files. Therefore, for the rest of the analysis, we only included the 39 hospitals that have the required information in their machine-readable files on their websites. We created an Excel file that included those 39 hospitals along with the 4 services—CPT 45380, 29827 and 70553 and DRG 807—mentioned in the literature ( Table 1 ) for 4 popular (suggested by Florida Blue experts) commercial carriers (Health Maintenance Organization [HMO] or Preferred Provider Organization [PPO] plans)—Aetna, Florida Blue, Cigna, and UnitedHealthcare.

Participants were recruited for the pilot and randomly assigned by the Qualtrics XM platform to answer multiple-choice questions and fill in blanks based on the given scenarios. First, participants responded to questions regarding the awareness of PT and then were divided into 2 groups randomly to answer questions regarding the usability of hospital-provided PT information. One group was assigned hospitals’ website links (control group), while the other group was given an Excel file with the same information provided in files on hospitals’ websites, but in a manner that was designed to allow easier comparison of prices across hospitals ( Multimedia Appendix 1 ). Participants were given 3 scenarios that asked them to find a procedure’s price based on their hospital and insurer selection to compare hospital-provided information with Excel. We provide some examples of hospitals’ files and our Excel file in Multimedia Appendix 1 and the survey experiment questions in Multimedia Appendix 2 .

Value for Machines: Schema Integration—Machine-Readable Files Representation

Through meticulous investigation of machine-readable files from 39 hospitals, we discovered that these files may vary in formats such as CSV or JSON, posing a challenge for machines to effectively manage the data within these files. Another significant obstacle arises from the lack of uniformity in data representation across these files, rendering them unsuitable for machine use without a cohesive system capable of processing them collectively. Our analysis revealed that hospitals within a single health system exhibit consistent data representation, although service prices may differ (we include both the same and different chargemaster prices in our study), while substantial disparities in data representation exist between hospitals affiliated with different health systems.

Moving forward, we will use the terms “data representation” and “schema” interchangeably, with “schema” denoting its database management context. In this context, a schema serves as a blueprint outlining the structure, organization, and relationships of data within a database system. It encompasses key details such as tables, fields, data types, and constraints that define the stored data. To systematically illustrate schema differences among hospitals associated with different health systems, we adopted the methodology outlined in reference [ 27 ] for schema integration, which offers a valid approach for comparing distinct data representations. The concept of schema integration encompasses four common categories: (1) identical: hospitals within the same health system adhere to this concept as their representations are identical; (2) equivalent: while hospitals in health system “A” may present different representations from those in health system “B,” they possess interchangeable columns; (3) compatible: in cases where hospitals across different health systems are neither identical nor equivalent, the modeling constructs, designer perception, and integrity constraints do not contradict one another; and (4) incompatible: in situations where hospitals within different health systems demonstrate contradictory representations, distinct columns exist for each health system due to specification incoherence.

Our analysis focused on health systems in Florida that encompassed a minimum of 4 main hospitals, using the most up-to-date data available on their respective websites. Within this scope, we identified 8 health systems with at least 4 main hospitals, of which 88% (7/8) of health systems had published machine-readable files on their websites. Consequently, our analysis included 65% (36/55) of hospitals that possessed machine-readable files available on their websites. To facilitate further investigation by interested researchers, we have made the analyzed data accessible on a cloud-based platform. During our analysis, we meticulously extracted the schema of each health system by closely scrutinizing the hospitals associated with each health system, capturing key details such as tables, fields, and data types. Subsequently, we compiled a comprehensive master field name table trying to have the same data type and field names that make it easier for machines to retrieve information. We elaborate on the master field names table in greater detail within the results section.

Value for Users

Question 1 (pt awareness).

Based on the responses, it is evident that participants need to be made aware of the PT legislation. Among the participants, 64% (49/76) reported that they had not heard about the legislation. However, they believe it is important to know the service price before receiving it—response charts are provided in Multimedia Appendix 3 .

Question 2 (Human Usability of PT Information)

Based on the responses to scenarios, the average number of correct responses is not equal between the 2 groups, that is, the treatment group (mean 1.23, SD 1.30) found more correct answers than the control group (mean 2.76, SD 0.58; t 65 =6.46; P <.049; d =1.52). The t tests (2-tailed) for the other questions in the experiment are in Multimedia Appendix 4 .

These suggest that current files on hospitals’ websites are not consumer-friendly, and participants find it challenging to estimate out-of-pocket costs for a desired service. For this reason, in addition to making the files easier to use, this information should also include thorough documentation that explains what each column represents, up to what amount an insurer covers for a specific service, or the stated price covers up to how many days of a particular service, that is, “contracting method.” For example, based on consulting with one of the senior network analysts of Florida Blue, some prices for a service like DRG 807 are presented as per diem costs, and based on the current information on these files, it cannot be recognizable without having comprehensive documentation for them.

Value for Machines

After carefully reviewing all machine-readable file schemas, we create a master field name table, including the available field names in machine-readable files ( Table 2 ). According to Table 2 , the first column represents master field names that we came up with, and the following columns each represent hospitals within a health system. The “✓” mark shows that hospitals within a health system have identical field names as we consider as master field names and the “written” cells show equivalent field names, meaning that hospitals within that health system use different field names—we write what they use in their representation—while the content is equivalent to what we select as the master field name. The “❋” mark means that although hospitals within health system #2 provide insurer names and plans in their field names, some codes make those columns unusable for machines to recognize them the same as master field names. We also include the type of field names for all representations in parentheses.

a As noted previously, since we focus on the health system level instead of the hospital level, our schema does not have hospital-level information; however, it would be beneficial to add hospital information to the table.

b ✓: it means the given master field name in that row appears on the given health system file in that column.

c str: shows “string” as the data type.

d int: shows “integer” as the data type.

e CPT: Current Procedural Terminology.

f HCPCS: Health care Common Procedure Coding System.

g Not applicable.

h Apr: all patients refined.

i DRG: Diagnosis-Related Group.

j Ms: Medicare severity.

k CDM: charge description master.

l UB: uniform billing.

m float: it shows “float” as the data type.

n ❋: it means that although hospitals within health system #2 provide insurer names and plans in their field names, some codes make those columns unusable for machines to recognize them the same as master field names.

We did reverse engineering and drew entity-relationship diagrams (ERDs) for each hospital based on their data representation. However, as hospitals within the same health system have the same ERDs, we only include 1 ERD for each health system ( Figure 1 ). According to Figure 1 , although hospitals have tried to follow an intuitive structure, we can still separate them into three groups: (1) group I: all hospitals within this group have several columns for different insurers. As shown in the ERDs, we decided to have a separate entity, called “Insurance” for this group; (2) group II: all hospitals within this group have many sheets, and each sheet belongs to a specific insurer with a specific plan. As shown in the ERDs, we decided to create an “Insurance_Name” entity for this group’s ERD to show the difference in data representation; and (3) group III: all hospitals within this system have a “payer” column which includes the names of insurers without their plans. As shown in the ERDs, we decided to put this column as an attribute in the “Service” entity, and do not have an “Insurance” entity for this group’s ERD.

In conclusion, although most hospitals have adopted group I logic for data representation, for full similarity, a standard representation with the same intuitive field names (like what we suggest as the master field name; Table 2 ) should be proposed so that it can cover all systems’ data representations and be used as machine-readable file, for at least machine benefits. Mainly, standardization in the format and semantics of the provided data can help substantially in making the data more machine friendly.

Comparison With New CMS Guidelines

Recently, CMS has published guidelines regarding the PT legislation [ 28 ]. The most recent CMS guideline is a step forward in ensuring standardization but is still only recommended and is not mandatory. These guidelines exhibit overlaps with our fields in Table 2 , with slight differences attributed to granularities. Our observation reveals that hospitals within the same health system adopt a uniform schema. Therefore, our suggested schema operates on the granularity of health systems rather than individual hospitals.

The recent CMS guidelines allocate 24% (6/25) of field names specifically to hospital information, encompassing details such as “Hospital Name,” “Hospital File Date,” “Version,” “Hospital Location,” “Hospital Financial Aid Policy,” and “Hospital Licensure Information.” These details, absent in current hospital files, are crucial for informed decision-making. As noted previously, since we focus on the health system level instead of the hospital level, our schema does not have hospital-level information; however, it would be beneficial to add hospital information to the tables.

Our analysis reveals that the 11 field names in Table 2 align with the field names in the new CMS guidelines, demonstrating a substantial overlap of 58% (11/19). The corresponding CMS field names (compatible with our schema) include “Item or Service Description (Description or CDM Service Description),” “Code (Code),” “Code Type (Code Type),” “Setting (Patient Class),” “Gross Charge (Gross Charge),” “Discounted Cash Price (Discounted Cash Price),” “Payer Name (Insurer Name),” “Plan Name (Insurer Plan),” “Payer Specific Negotiated Charge: Dollar Amount (Price),” “De-identified Minimum Negotiated Charge (Min Negotiated Rate),” and “De-identified Maximum Negotiated Charge (Max Negotiated Rate).” Additionally, both our schema and the new CMS guidelines propose data types for each field name.

In our schema, which represents current hospitals’ files, there are 5 field names absent in the new CMS guidelines “Revenue Description,” “Revenue Code,” “Package/Line Level,” “Procedure ID,” and “Self Pay.” Conversely, the new CMS guidelines introduce 8 additional field names “Billing Class,” “Drug Unit of Measurement,” “Drug Type of Measurement,” “Modifiers,” “Payer Specific Negotiated Charge: Percentage,” “Contracting Method,” “Additional Generic Notes,” and “Additional Payer-Specific Notes.” We regard these new field names as providing further detailed information and enhancing consumer decision-making. If hospitals within a health system adopt consistent formats and can map their formats to the new CMS guidelines clearly in a mapping document they also provide, this can be more useful than the current optional guideline that is suggested.

In summary, since our analysis is based on the current data schema that hospitals have in place, we believe the schema we put out is easier to implement with minimal change to what the hospitals are currently doing. However, given the recent CMS guidelines, we recommend adding 8 additional fields as well as hospital-specific information.

Implications

The PT legislation aims to enable informed decision-making, reduce out-of-pocket expenses, and decrease overall health care expenditures. This study investigates the usage of current files by individuals and machines. Our results, unfortunately, suggest that PT data—as currently reported—appear to be neither useful for patients nor machines, raising important questions as to what these appear to be achieving today. Moreover, the findings indicate that even individuals with basic computer knowledge struggle with the usability of these files, highlighting the need for significant revisions to make them consumer-friendly and accessible to individuals of all technical proficiency levels. Additionally, inconsistencies in data representation between hospitals affiliated with different health systems pose challenges for machines, necessitating schema design improvements and the implementation of a standardized data representation. By addressing these concerns, PT legislation can achieve consistency and enhance machine readability, thus improving its effectiveness in promoting informed decision-making and reducing health care costs.

Although the official announcement of PT legislation is recent, prior studies [ 15 - 17 ] have attempted to evaluate the usability of PT, while subsequent studies [ 19 - 22 ] have examined the effectiveness of PT tools following the announcement. However, despite the introduction of PT rules, it appears that the usability of these files has not undergone significant improvements, indicating the necessity for proactive measures from responsible executives to ensure the effectiveness of this legislation. Our analysis of this matter emphasizes 2 primary factors—a lack of awareness among stakeholders and the challenges associated with using files due to inconsistencies in their format and representation.

As of April 2023, the CMS has issued over 730 warning notices and 269 requests for Corrective Action Plans. A total of 4 hospitals have faced Civil Monetary Penalties for noncompliance, and these penalties are publicly disclosed on the CMS website. The remaining hospitals subjected to comprehensive compliance reviews have either rectified their deficiencies or are actively engaged in doing so. While we acknowledge these efforts to comply with PT rules, our research revealed a notable disparity in data representation among hospitals affiliated with different health systems. Consequently, we focused on schema design and proposed the implementation of a master field name that encompasses a comprehensive data representation derived from an analysis of 36 hospitals. Standardizing the data representation across all health systems’ machine-readable files will effectively address concerns about consistency. Therefore, significant modifications are required for the PT legislation to enhance machine readability and provide clearer guidance on the design and structure of the files’ schema. If the hospital-provided information is consistent and of high quality, PT tools provided by health insurers may be able to estimate an individual’s total expenses more accurately.

Limitations

Our objective was to have an equal number in both groups. However, in the case of the group tasked with obtaining information from the hospitals’ websites, most did not finish the task and dropped out without completing it. This occurred because the task of retrieving the cost from the hospitals’ websites in its current form is complex, as indicated by feedback from some participants. Only 19% (13/67) completed the task in that group (control group). Although this is a limitation of the study, it also highlights the complexity of obtaining cost information from hospitals’ websites in the current form. In the treatment group, 81% (54 out of 67) of participants completed the task of retrieving the data, and the completion percentage was much higher.

Conclusions

Due to the poor usability and inconsistency of the formats, we, unfortunately, did not find evidence that the PT rule as implemented currently is useful to consumers, researchers, or policy makers (despite the legislation’s goals that files are “consumer-friendly” and “machine-readable”). As 1 solution, we suggest a master field name for the data representation of machine-readable files to make them consistent, at least for the machines. Building tools that enable customers to estimate out-of-pocket costs is facilitated by having consistent machine-readable files across all health systems, which can be considered as future work for researchers and companies to help the PT rule reach its main goal, which is providing useful information for patients and reducing health care expenditures. In addition, another worthwhile approach to reducing some of the exorbitant health care costs in the United States would be to integrate clinical decision support tools into the providers’ workflow, triggered by orders for medications, diagnostic testing, and other billable services. In this regard, Bouayad et al [ 29 ] conducted experiments with physicians to demonstrate that PT, when included as part of the system they interact with, such as clinical decision support integrated into electronic health record systems, can significantly aid in cost reduction. This is a promising direction for practice but needs to be implemented carefully to avoid unanticipated consequences, such as scenarios where cost is incorrectly viewed as a proxy for quality, or where the use of this information introduces new biases for physicians and patients.

Conflicts of Interest

None declared.

Example of Excel format of hospitals’ files and our created Excel file.

Survey questions and experiment scenarios.

Participants’ responses chart regarding price transparency awareness.

The t test analysis regarding human usability of price transparency information based on participants’ responses.

• McGough M, Winger A, Rakshit S, Amin K. How has U.S. spending on healthcare changed over time? Health System Tracker. 2022. URL: https://www.healthsystemtracker.org/chart-collection/u-s-spending-healthcare-changed-time/ [accessed 2024-03-13]
• Christensen HB, Floyd E, Maffett M. The only prescription is transparency: the effect of charge-price-transparency regulation on healthcare prices. Manag Sci. 2020;66(7):2861-2882. [ CrossRef ]
• Muir MA, Alessi SA, King JS. Clarifying costs: can increased price transparency reduce healthcare spending? UC Hastings Research Paper No. 38 (SSRN). Feb 26, 2013.:319-367. [ FREE Full text ] [ CrossRef ]
• Reinhardt UE. Health care price transparency and economic theory. JAMA. 2014;312(16):1642-1643. [ CrossRef ] [ Medline ]
• CY 2020 hospital Outpatient Prospective Payment System (OPPS) policy changes: hospital price transparency requirements (CMS-1717-F2). CMS.gov. 2020. URL: https://tinyurl.com/mrafxtvd [accessed 2024-03-13]
• Secretary Azar statement on proposed rule for hospital price transparency. HHS.gov. 2020. URL: https://tinyurl.com/yc4dx2vx [accessed 2024-03-13]
• Mehrotra A, Brannen T, Sinaiko AD. Use patterns of a state health care price transparency web site: what do patients shop for? Inquiry. 2014;51:0046958014561496. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai S, Hatfield LA, Hicks AL, Sinaiko AD, Chernew ME, Cowling D, et al. Offering a price transparency tool did not reduce overall spending among California public employees and retirees. Health Aff (Millwood). 2017;36(8):1401-1407. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sinaiko AD, Joynt Maddox KE, Rosenthal MB. Association between viewing health care price information and choice of health care facility. JAMA Intern Med. 2016;176(12):1868-1870. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai S, Hatfield LA, Hicks AL, Chernew ME, Mehrotra A. Association between availability of a price transparency tool and outpatient spending. JAMA. 2016;315(17):1874-1881. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sinaiko AD, Rosenthal MB. Examining a health care price transparency tool: who uses it, and how they shop for care. Health Aff (Millwood). 2016;35(4):662-670. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai SM, Shambhu S, Mehrotra A. Online advertising increased New Hampshire residents' use of provider price tool but not use of lower-price providers. Health Aff (Millwood). 2021;40(3):521-528. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kona M, Corlette S. Hospital and insurer price transparency rules now in effect but compliance is still far away. Health Affairs Forefront. 2022. URL: hhttps://tinyurl.com/3x6ymxf2 [accessed 2024-03-13]
• Wheeler C, Taylor R. New year, new CMS price transparency rule for hospitals. Health Affairs Forefront. 2021. URL: https://www.healthaffairs.org/content/forefront/new-year-new-cms-price-transparency-rule-hospitals [accessed 2024-03-13]
• Chernew M, Cooper Z, Larsen-Hallock E, Morton FS. Are health care services shoppable? Evidence from the consumption of lower-limb MRI scans. National Bureau of Economic Research. 2021. URL: https://www.nber.org/papers/w24869 [accessed 2024-03-13]
• Mehrotra A, Dean KM, Sinaiko AD, Sood N. Americans support price shopping for health care, but few actually seek out price information. Health Aff (Millwood). 2017;36(8):1392-1400. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brown ZY. Equilibrium effects of health care price information. Rev Econ Stat. 2019;101(4):699-712. [ FREE Full text ] [ CrossRef ]
• Wu SJ, Sylwestrzak G, Shah C, DeVries A. Price transparency for MRIs increased use of less costly providers and triggered provider competition. Health Aff (Millwood). 2014;33(8):1391-1398. [ CrossRef ] [ Medline ]
• Glover M, Whorms DS, Singh R, Almeida RR, Prabhakar AM, Saini S, et al. A radiology-focused analysis of transparency and usability of top U.S. hospitals' chargemasters. Acad Radiol. 2020;27(11):1603-1607. [ CrossRef ] [ Medline ]
• Hayatghaibi SE, Alves VV, Ayyala RS, Dillman JR, Trout AT. Transparency and variability in pricing for pediatric outpatient imaging in US children's hospitals. JAMA Netw Open. 2022;5(3):e220736. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Arvisais-Anhalt S, McDonald S, Park JY, Kapinos K, Lehmann CU, Basit M. Survey of hospital chargemaster transparency. Appl Clin Inform. 2021;12(2):391-398. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reddy S, Daly G, Baban S, Kadesh A, Block AE, Grimes CL. Accessibility and usability of hospital chargemasters in New York state. J Gen Intern Med. 2022;37(8):2130-2131. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Robinson JC. Hospital quality competition and the economics of imperfect information. Milbank Q. 1988;66(3):465-481. [ Medline ]
• Patel KN, Mazurenko O, Ford E. Analysis of hospital quality measures and web-based chargemasters, 2019: cross-sectional study. JMIR Form Res. 2021;5(8):e26887. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Batty M, Ippolito B. Mystery of the chargemaster: examining the role of hospital list prices in what patients actually pay. Health Aff (Millwood). 2017;36(4):689-696. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Muhlestein DB, Wilks CEA, Richter JP. Limited use of price and quality advertising among American hospitals. J Med Internet Res. 2013;15(8):e185. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Batini C, Lenzerini M, Navathe SB. A comparative analysis of methodologies for database schema integration. ACM Comput Surv. 1986;18(4):323-364. [ FREE Full text ] [ CrossRef ]
• Voluntary hospital price transparency machine-readable file sample format data dictionary (version 1.1). CMS.gov. URL: https:/​/www.​cms.gov/​files/​document/​hospital-price-transparency-machine-readable-data-dictionary-tall.​pdf [accessed 2024-03-13]
• Bouayad L, Padmanabhan B, Chari K. Can recommender systems reduce healthcare costs? the role of time pressure and cost transparency in prescription choice. MIS Q. 2020;44(4):1859-1903. [ CrossRef ]

Abbreviations

Edited by S He; submitted 07.07.23; peer-reviewed by KN Patel, R Marshall, G Deckard; comments to author 03.12.23; revised version received 21.01.24; accepted 26.02.24; published 29.03.24.

©Negar Maleki, Balaji Padmanabhan, Kaushik Dutta. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 29.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.