Comments on Benefit-Risk Factors to Consider When Determining Substantial Equivalence in Premarket Notifications

October 14, 2014

Division of Dockets Management (HFA-305)
Food and Drug Administration
5600 Fishers Lane, Room 1061
Rockville, MD 20852

Comments of Members of the Patient, Consumer, Public Health Coalition
on the Draft Guidance
“Benefit-Risk Factors to Consider When Determining Substantial Equivalence in Premarket Notifications [510(k)] with Different Technological Characteristics”
[Docket No. FDA-2014-D-0900]

As members of the Patient, Consumer, and Public Health Coalition, we strongly oppose the draft guidance Benefit-Risk Factors to Consider When Determining Substantial Equivalence in Premarket Notifications [510(k)] with Different Technological Characteristics.  The three major shortcomings of the guidance are as follows:

  1. It focuses on benefits and risks that cannot be accurately measured in the absence of large, long-term, well-designed clinical trials.  Such studies are never available for new devices being considered for 510(k) clearance.  In fact, even small, short-term, well-designed clinical trials are almost never available.
  2. The wording of the guidance is vague and rather than clarifying the criteria for substantial equivalence, the guidance illustrates the very subjective decisions that the Food and Drug Administration (FDA) relies on.  The lack of consistent, clear standards is unfair to device companies and physicians and harmful to patients and the public health.
  3. The specifics regarding surrogate endpoints, emphasis on postmarket data, and patients’ perspectives on risk assessment would further reduce the already extremely limited safeguards available to ensure the safety and effectiveness of new medical devices cleared through the 510(k) process.


Principal Benefits and Risk Factors for Performance Data

This section states that the FDA will assess “on a case-by-case basis” what it may find acceptable in a substantial equivalence decisions. We oppose these subjective judgments made on case-by-case assessments because they do not foster a standardized approach.  The Center for Devices and Radiological Health (CDRH) should be able to make clear guidelines about what substantial equivalence means and how it needs to be proven scientifically.


Assessment of the Benefits of Devices

The FDA takes into account the following four factors in assessing information provided in a 510(k) submission: Type of benefits, magnitude of the benefits, probability of the patient experiencing one or more benefits, and the duration of effects.

Type of benefits

The draft guidance list of benefits includes patient health, patient satisfaction, quality of life, and probability of survival among other benefits.  It then adds that “these endpoints denoting clinical benefit…may be demonstrated by use of validated surrogate endpoints.”  The word “validated” is not defined.  The FDA should avoid using surrogate endpoints because the FDA’s judgment of what is validated has been very subjective.  Patients want improved health, relief from symptoms, longer life, and improved quality of life.  Even if surrogate endpoints correlate with any of those outcomes, the surrogate endpoints would need to be scientifically validated for specific demographic subgroups, such as women, men, Whites, Blacks, Hispanics, and people over 60.  The CDRH does not have a good track record on surrogate endpoints or subgroup analysis, unfortunately.

Magnitude of the benefits

The draft guidance states, “The change in clinical study subjects’ condition or clinical management” allows the FDA to assess the benefits patients receive from the treatment. However, a recent study showed that only about 6% of 510(k) implant applications—including implants with different technological characteristics than the predicate device—mention the existence of clinical data in their summaries, whereas many more mention the use of non-clinical data.[1]  If the FDA is using clinical data as the basis of substantial equivalence decisions, the law requires that information to be publicly available “in sufficient detail.”[2]

Probability of the patient experiencing benefits

The guidance does not explain the evidence that will be used to determine patient benefits.  Since 510(k) applications so rarely include any clinical data, and those that are included tend to be a small study, it is misleading to claim that the FDA can determine the “[d]emonstration of a large benefit experienced by a small proportion of subjects [which] may raise considerations that differ from those in instance where a small benefit is experienced by large proportion of subjects.”  The key question is, given the small or non-existent clinical studies of new devices included in virtually all 510(k) applications, how can the proportion of subjects who experience a large benefit be identified or targeted? This would require that accurate diagnostics be available to identify populations that would benefit the most from the device, which is often not the case.

Regarding the assessment of probable risks and harms by reviewing factors individually and in the aggregate, we agree that “When multiple harmful events occur at once, they have a greater aggregate effect.”  However, we are concerned that the draft guidance use of “number and rates” of harmful events associated with a device is not practical, since it is commonly known that Manufacturer and User Facility Device Experience (MAUDE) undercounts adverse events.  The same can be said for device-related non-serious adverse events and procedure-related complications.  In the absence of clinical trials or registry data for the vast majority of 510(k) devices, CDRH does not have accurate reporting mechanisms currently functioning to accurately reflect these numbers. The FDA is still years away from using unique device identifiers (UDIs) and device registries on enough devices to accurately measure adverse events.

Regarding the probability of a harmful event, the draft guidance states, “FDA would factor whether an event occurs once or repeatedly into the measurement of probability.” In the absence of long-term clinical trials, accurate measurement of repeated harmful events is not possible.  Even if it were possible, it would not weigh the seriousness of an event. For example, death happens only once.  The draft guidance also states that the FDA may be able to predict which patients “may experience a harmful event.” Again, this seems unlikely given that very few 510(k) devices have clinical trial data and those that do usually have relatively few patients, little diversity, and no subgroup analyses.

Duration of effects

The draft guidance states that some devices can cause temporary or minor harms, others can cause repeated but reversible harm, and others can cause permanent, debilitating injury.  The FDA plans to consider the severity and duration of the harm. Again, in the absence of clinical trials or more accurate adverse reporting systems, the FDA is not able to accurately evaluate the likelihood or severity of harms. Moreover, the FDA and patients may have different definitions for “reversible harm.” For example, when metal-on-metal hips were recalled, the FDA classified the recall as Class II, “moderate risk” recalls.  And yet, removal of those implants required multi-hour surgery that could cause permanent harm or death, and revision hip surgery almost always leaves patients with disabilities or limited abilities compared to the first hip surgery.  A failed revision surgery can result in patients losing mobility and thus having to move from independent living to a nursing home.


Patient tolerance for risk

The draft guidance states that the FDA may consider the patient’s perspective and notes that some patients are “willing to accept a higher level of risk to achieve a higher probable benefit.” There are two major fallacies with that statement:

  1. Patients may say they are willing to accept a higher risk, but when they have a negative outcome they inevitably wonder why the FDA allowed such an unsafe product on the market, and why this terrible harm happened to them.
  2. A 510(k) device is cleared for everyone, not just those willing to accept a higher risk. Also, the device can then be used as a predicate for a new device, which will also be used by patients who did not choose to accept the high level of risks.  By catering to those patients who say they are willing to accept higher risks, the FDA by default makes all patients who may use the device accept higher risks.


Risk mitigation

The draft guidance states that if a new device has an increased risk, the FDA may approve it, “if the risk is appropriately mitigated.” The draft guidance recommends “appropriate information within labeling (e.g., warnings, precautions, contraindications)” to mitigate risks. What evidence does the FDA have to conclude that health care professionals read the labels, are influenced by warnings on them, or share that information with their patients?  On the contrary, in our work with thousands of patients, most tell us that their physicians did not ask even simple questions about allergies or illnesses that were listed on the label as contraindications.  Our work with physicians has clearly shown that many do not read device labels and that surgeons often do not ever see the labels on the devices they use in the operating room.


Degree of uncertainty and innovative technology

We oppose the statement, “FDA may accept a greater degree of premarket uncertainty regarding a device’s benefit-risk profile through a greater reliance on postmarket controls…if FDA’s overall assessment is sufficiently balanced by other factors…”  This is too vague to provide meaningful guidance. Moreover, it is the FDA’s mission to provide reasonable assurance of safety and effectiveness before a device is put on the market, not to wait years after millions of patients may have used it.  A recent review of postapproval studies for medical devices found that “small sample sizes, delays in reaching protocol agreement, and lack of availability of findings may hinder their ability to be clinically useful.”[3]

Regarding innovative technology, the draft guidance states that the FDA “may accept greater uncertainty” in assessing benefits and risks compared to the predicate device when the technological improvements are “important for public health.”  We oppose this statement because the phrase “important for public health” is too vague and subjective, especially when combined with the other statement in this section that the FDA will evaluate innovative changes on a “case-by-case basis.”



We oppose the draft guidance as written and strongly urge the FDA to rewrite it.  The current draft guidance relies on vague wording and subjective judgments, not science or technological assessments.  When it is more specific, it promotes the use of surrogate endpoints and the risk tolerance of a small number of patients whose views come to the FDA’s attention.

American Medical Student Association
American Medical Women’s Association
Annie Appleseed Project
Breast Cancer Action
Center for Medical Consumers
Connecticut Center for Patient Safety
National Center for Health Research
National Consumers League
The TMJ Association

The Patient, Consumer, and Public Health Coalition can be reached through Paul Brown at 202-223-4000 or

[1] Zuckerman D, Brown P, Das A, (September 29, 2014). Lack of Publicly Available Scientific Evidence on the Safety and Effectiveness Implanted Medical Devices.  JAMA Intern Med.

[2] US Food and Drug Administration. Content and format of a 510(k) summary. 21CFR§807.92 Accessed October 9, 2014.

[3] Reynolds IS, Rising JP, Coukell AJ, Paulson KH and Redberg RF (September 292104). Assessing the Safety and Effectiveness of Devices After US Food and Drug Administration Approval. JAMA Intern Med.