The Role of Machine Learning in the Fight Against America’s Opioid Epidemic
In a recent blog post, one of our staff physicians, Dr. Gus Crothers, discussed the role of prevention in combating the opioid epidemic in America. To recap: the best way of preventing opioid overdose deaths is to reduce exposure to these drugs by improving opioid prescribing practices. At Grand Rounds, we aren’t able to directly impact the prescribing practices of any single doctor, but we do have control over which health care providers we recommend to our members. For example, we recently built a machine learning model to identify which primary care physicians (PCP) have unsafe prescription drug prescribing patterns.
At its core, a machine learning algorithm uses a set of input variables (features) to predict one or more output variables (labels). In our case, the labels are indicators for whether or not a physician has evidence of dangerous prescribing patterns in their claims data. We consulted with our medical team to define metrics upon which we could rank the prescribing patterns of doctors. The metric used in our model calculates the number of patients receiving a dangerously high-dose opioid prescription* from a physician, relative to the total number of patients receiving opioid prescriptions from that physician. This metric does not attempt to judge whether or not opioids are indicated for any given patient (this is very difficult to assess from claims), but instead penalizes physicians if they are consistently prescribing dangerous doses when they decide that opioids are necessary. Since the scope of this model is limited to PCPs, the number of patients prescribed justifiably high doses (for example, oncology patients) is small. We found that, of physicians who prescribe opioids to their patients, more than 80% have written at least one dangerously high-dose prescription. Furthermore, roughly 20% of these physicians have at least 1 in 5 of their opioid patients taking high doses.
If we had access to comprehensive prescribing data on all physicians, we could simply rank them based on this metric and tag the most dangerous offenders as having bad prescription practices. However, claims data are rarely provider complete, meaning that we may not have information about every prescription written by every provider. In cases where data on a physician’s opioid prescribing patterns are incomplete, we use our machine learning model to predict whether or not they are likely to have safe opioid prescribing patterns. Our model uses more general features about a physician’s drug prescribing patterns to make these predictions, including but not limited to:
- The average days supply across all prescriptions;
- The average number of refills authorized across all prescriptions; and
- The fraction of prescriptions belonging to each DEA schedule.
To train our model, we first applied positive labels to PCPs with a large number of opioid patients and a large fraction of those patients on high-dose opioids. We also applied negative labels to a subset of all other PCPs. Our model was then able to learn which general prescribing features are predictive of dangerous opioid prescribing patterns. Once the model was trained, we used it to predict whether or not a PCP has safe opioid prescribing practices, even in the absence of any claims data about their opioid prescribing patterns.
Validation of our model from a medical perspective
We looked at the feature importances of the input variables to explore how our model makes predictions, and whether or not the important features make sense. The features most predictive of opioid prescribing patterns are listed below.
Associated with safer prescribing patterns
1. A higher number of prescriptions written per patient
2. Larger fractions of patients prescribed schedule IV drugs, relative to the patient
fractions prescribed more strictly controlled drugs
Associated with more dangerous prescribing patterns:
1. A large fraction of high dose benzodiazepine patients (relative to all benzodiazepine patients)
2. A large number of DEA schedule II drugs per patient.
The first of the features associated with safer prescribing patterns is slightly counterintuitive, but is likely a result of safe doctors prescribing milder medications first, and only escalating to opioids as absolutely necessary. This behavior would result in a larger prescription count per patient, and is consistent with expert recommendations for appropriate opioid prescribing.
The first feature associated with more dangerous prescribing patterns is interesting because benzodiazepines are another type of drug responsible for a significant number of overdose deaths each year. These drugs are especially dangerous when combined with opioids. We looked in our claims data for evidence of PCPs prescribing overlapping opioid-benzodiazepine prescriptions, and found that of the PCPs tagged as having bad prescription practices by our model, at least** two-thirds of them have prescribed benzodiazepines concurrently with opioids (or vice-versa). This prescribing pattern is known to dramatically increase the risk of opioid overdose or an adverse event, and is discouraged by expert recommendations for safe prescribing practices. Clearly, patients should have safer alternatives to these PCPs.
There are many factors beyond opioid prescribing that need to be considered when assessing physician quality. However, Grand Rounds is first and foremost committed to the safety of our members. We firmly believe that how and when a PCP chooses to prescribe an opioid has dramatic effects on patient safety. Through our bad prescription model, we are proactively fighting the opioid epidemic on our members’ behalf by reducing the chance that they will be exposed to dangerous prescribing practices. This model is used by our Matching Algorithm along with other quality measures to ensure that members searching for new physicians have access to the safest, most appropriate care in their area.
* The threshold for an opioid prescription to be considered a dangerously high dose is 90 morphine milligram equivalent (MME) per day. The CDC recommends that physicians “use extra precautions when increasing to >=50 MME/day” and “avoid or carefully justify increasing dosage to >= 90 MME/day”. Source: https://www.cdc.gov/drugoverdose/pdf/calculating_total_daily_dose-a.pdf
** As mentioned, claims data are rarely provider complete, so the actual number may be higher