A large dataset for studying the early readmission of diabetic patients problem
Oscar Reyes1, Krzysztof J. Cios4,5, and Sebastián Ventura1,2,3*
Abstract
Considering the growing prevalence of diabetes in the global population, the prediction of the early readmission of patients is a very important and demanding task for healthcare payers and hospitals. This paper presents and describes a large dataset of 101,766 records collected from 130 U.S.A hospitals over a 10-year period. The dataset was mainly designed to study the problem of early readmission of diabetic patients, but it is also relevant for studying important data relationships describing diabetic patients. In this work, the process followed to extract the dataset from its original sources and the performed preprocessing steps are described. Several statistics and data metrics are also measured on the dataset to describe and characterize its complexity. Furthermore, several classifiers belonging to different learning paradigms are assessed on the dataset, and the overall importance of the predictive variables is estimated. This work will help researchers to better understand this dataset, the existing data relationships, and the complexity of its underlying classification task.
1Department of Computer Science and Numerical Analysis, University of Córdoba, 14071 Córdoba, Spain. 2Knowledge Discovery and Intelligent Systems in Biomedicine Laboratory, Maimonides Biomedical Research Institute of Córdoba, 14004 Córdoba, Spain. 3Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia. 4Department of Medicine, Virginia Commonwealth University, Richmond, VA 23298, USA. 4Department of Computer Science, Virginia Commonwealth University, Richmond, VA 23284, USA. 5IITiS Polish Academy of Sciences, 44-100 Gliwice, Poland.
Payment penalties based on the quality of care metrics has implied continuous substantial changes in the business models of the healthcare systems (Futoma, Morris, and Lucas 2015). For example, the early readmission penalties regulated in various countries have provoked the need of reliable patient-care strategies that allow hospital systems identify, at the time of discharge, those patients who are at high risk of being readmitted within a short period of time (typically within 30 days after the original admission) (Leppin et al. 2014; Verhaegh et al. 2014; Golmohammadi and Radnia 2016). In this same context, diabetes is nowadays a major health problem of prevalence in developed countries, wherein patients commonly suffer cardiovascular and other systemic complications, producing high morbidity and mortality rates (Gahete et al. 2018). Accordingly, the problem of minimizing the early readmissions of diabetic patients has gained an increasing attention in the last years (Leppin et al. 2014; Strack et al. 2014), considering the growing number of persons with diabetes type II, and the large proportion of healthcare resources that this illness consumes.
The aforementioned necessity has lead to several works and datasets focusing on studying the early readmission of diabetic patients. For example, Jiang et al. (Jiang et al. 2003) studied how the demographic and socioeconomic factors could have a relevant influence on readmission rates. Garrison et al. (Garrison, Katon, and Richardson 2005) described the impact of psychiatric comorbidities on readmissions in youth. Dugan (Dungan 2012) analyzed several general measures and diabetes-specific interventions to prevent the readmissions. Healy et al. (Healy et al. 2013) explored the relationship between inpatient diabetes education and hospital readmissions in patients with poorly controlled diabetes. Eby et al. (Eby et al. 2014) studied the readmission risk on a dataset of more than 52,000 patients, and they showed the importance of the escalation of diabetic treatments, especially those escalated from having no records of anti-diabetic medications to treatment with insulin. Rubin (Rubin 2015) determined that inpatient education, better discharge instructions, coordination of care and post-discharge support are potential ways to reduce readmission risk. Finally, Rubin et al. (Rubin et al. 2016) developed and validated a tool on a cohort of 44,203 records that predicts readmission risk of diabetic patients, and later they further assessed their study in (Rubin et al. 2017), where a cohort of 8,189 records of diabetic patients with cardiovascular disease was analyzed.
In this work, a large and potential dataset focusing in the early readmission of diabetic patients problem is described and analyzed. This dataset was extracted from the HIPAA-compliant Health Facts database (Cerner Corporation, Kansas City, MO), collecting clinical records from 130 hospitals throughout the U.S.A. The database comprises data systematically collected from electronic medical records and includes a wide variety of elements, such as encounter records, demographic data, diagnoses and in-hospital procedures documented by ICD9 codes, pharmacy data, in-hospital mortality and hospital features. All data were de-identified in compliance with the Health Insurance Portability and Accountability Act of 1996 before being provided to the investigators.
An early version of this dataset was previously presented and analyzed by our team in (Strack et al. 2014), but at that time the dataset only contained 70,000 records of diabetic patients, and that study aimed at examining the relevance of HbA1C measurements, by conducting simple statistical analysis, to improve patient outcomes and decrease readmission frequencies. In this work, however, the dataset is expanded to 101,766 records, and an extensive description of the properties, data relationships and complexity of this dataset is provided. Therefore, this work presents a further description of the characteristics of this large data collection, allowing to characterize the difficulty of this dataset in conducting an automatic classification task; the prediction task consist on estimating whether a patient will be readmitted to the hospital in 30 days or less. The study was carried out first by means of conducting a data description and analysis which characterized the existing data relationships and measured the complexity of the dataset, and then by assessing a broad set of well-known classification algorithms of different learning paradigms; penalized linear models and non-linear models were considered. Also, all the considered classifiers have a built-in feature selection process, enabling the estimation of the overall relevance of the predictive variables describing the dataset.
This section describes all the materials and methods utilized to perform the collection, exploration and analysis of data. First, the process followed to extract the data collection from the original sources is portrayed, and then the actions carried out in data preprocessing phase are briefly described. Second, several statistics and data metrics that were used to characterize the complexity of the dataset and its underlying classification problem are explained. Finally, the set of classifiers that were assessed on the dataset are enumerated, and then it is explained how the predictive performance of these classifiers was estimated.
The dataset was extracted from the HIPAA-compliant Health Facts database, which consists of 41 tables in a fact-dimension schema having a total of 117 features. There were a total of 74,036,643 unique encounters corresponding to 17,880,231 unique patients, with 498,973 diabetic inpatients of any kind of diabetes; diabetic encounters are not all encounters of diabetic patients, but only those where diabetes was recognized as an existing health condition.
Information was filtered from the database for encounters satisfying the following criteria:
Only inpatient encounters (a hospital admission) were considered
Only diabetic encounters were analyzed, that is, one during which any kind of diabetes was entered to the system as a diagnosis
The length of patient’s stay was at least 1 day and at most 14 days
Laboratory tests were performed during the encounter
Medications were administered during the encounter
The following steps were conducted for preprocessing data:
Variables comprising unique identifiers for each record were removed, because they do not give any useful information for data analysis
Several steps were conducted to cope with missing values present in data
The levels of some categorical variables were reduced; this action was performed to reduce the impact in the classification algorithms of the nominal variables with many category levels
Scarce variables (e.g. several drugs that were never or scarcely prescribed) were discarded
A detection of outliers in the numeric variables was performed. Those observations that dropped outside of \(1.5 \times \textrm{IQR}\) (IQR -Inter Quartile Range- is the difference between the 75th and 25th quartiles) were considered as outliers, and then they were replaced with the median of the values of the encounters having the same type of readmission
Variables with variance equal or very close to zero were removed, since they do not give useful information for data analysis
Several steps on numeric variables were conducted: (I) the Yeo-Johnson power transformation (Y. and Richard 2000) was applied to stabilize variance, reducing the heteroscedasticity in data; (II) the variables were centred on their means aiming at focusing on the differences and not in the similarities in the data; (III) the variables were scaled dividing them by their standard deviations, thus all numeric predictors were equally important.
An analysis to discover possible relationships between the variables that describe the dataset was carried out, enabling a better understanding of data. The variable distributions with respect to the three levels defined by the target variable (variable representing the type of readmission) were studied. The Pearson’s \(\chi^2\) test was used to evaluate the independence between two dichotomous variables, whereas the Cramer’s V measure of association was employed for the case of nominal variables with more than two possible categories; the relationships considering all pairs of levels of two nominal variables were represented by mosaic plots with residual colors. The correlation between numeric and nominal variables was measured by means of using the correlation ratio \(\eta\) (eta), and the distributions of the numeric variables regarding the levels of the target variable were represented by conditional density plots. Furthermore, the Pearson product-moment was used to detect the correlations between pairs of numeric variables. Finally, the Mann-Whitney U test was conducted when testing the equality of the means of two groups of independent samples.
The difficulty of the dataset when solving the early readmission of diabetic patients problem was studied; in other words, the complexity of the underlying classification problem was characterized. The following three type of measures were considered to evaluate the geometrical complexity of the classification problem (Ho and Basu 2002; Luengo and Herrera 2012); a formal and complete definition of these measures can be consulted in (Orriols-Puig, Macia, and Ho 2010).
I) Measures of overlaps in the values of variables, which evaluate the capacity to separate samples of different classes. These measures can only be apply on numeric variables and, therefore, they were evaluated on the subset of encounters described by the numeric variables. In this group, the following four measures were used:
Maximum Fisher’s discriminant ratio (F1). F1 computes how spread the classes with respect to a specific variable are, by comparing the difference between class means with the sum of class variances; its possible values are in range \([0,+\infty]\)
Overlap of the per-class bounding boxes (F2). The overlap level of the tails of the distributions defined by the samples of each class is computed; values in range \([0,1]\)
Individual variable efficiency (F3). F3 represents the discriminative power of each individual variable; values in range \([0,1]\)
Collective variable efficiency (F4). F4 represents the discriminative power of all variables; values in range \([0,1]\).
II) Measures of class separability, which estimate whether the classes are separable or not by examining the length and linearity of the class boundary. In this group, the following three measures were considered:
Fraction of points on the class boundary (N1). N1 provides the percentage of nodes that link different classes in a minimum spanning tree constructed over the dataset; values in range \([0,1]\)
Ratio of average intra/inter class nearest neighbor distance (N2). For each sample, the minimum distances to neighbor samples belonging to the each class is computed; values in range \([0,+\infty]\)
Leave-one-out error rate of the one-nearest neighbor classifier (N3). N3 denotes how close samples belonging to different classes are in the feature space; values in range \([0,1]\).
III) Measures of geometry, topology, and density of manifolds, which provides an indirect characterization of the classes separability by assuming that the problem is composed of several manifolds spanned by each class. In this group, the following two measures were studied:
Fraction of maximum covering spheres (T1). Given an example, T1 measures how much a centred sphere can grow until an example of a different class is reached; values in range \([0,1]\)
Average number of points per dimension (T2). T2 calculates the ratio between the number of samples (\(n\)) in the dataset and the number of predictive variables, indicating the sparseness of the dataset; values in range \([0,n]\).
The performance of several classifiers was assessed on this dataset as a simple way to measure the complexity in predicting the early readmission of diabetic patients. All the classification algorithms have built-in feature selection mechanisms, allowing to estimate the importance of each predictive variable in the underlying classification task. The following 10 classifiers were assessed on the dataset:
Boosted Logistic Regression (BLR). BLR is a boosting algorithm formulated by Friedman et al. (Friedman, Hastie, and Tibshirani 2000), which can be viewed as an adaptation of AdaBoost algorithm (Freund and Schapire 1997) into a statistical framework. This algorithm leverages generalized additive models and applies the cost function of logistic regression; this last has extensively been used in the diabetes research (Kavakiotis et al. 2017), and in the early readmission problem (Wei et al. 2013).
C5.0 is an extension of the well-known C4.5 algorithm (Quinlan 1993). This builds decision trees using the concept of information entropy, and has been previously applied to the early readmission problem (Duggal et al. 2016a).
Classification And Regression Tree (CART). CART was proposed by Breiman et al. (Breiman et al. 1984), and it builds decision trees using different concepts of entropy than those used by C5.0. Mani et al. (Mani et al. 2012) used this algorithm in the prediction of the type 2 diabetes.
Conditional Inference Tree (CTree) is a unified framework that embeds recursive binary partitioning into the well defined theory of permutation tests (Hothorn, Hornik, and Zeileis 2006). This tries to solve some limitations that present other recursive partitioning algorithms like CART and C4.5.
Linear Discriminant Analysis (LDA) is a generalization of Fisher’s linear discriminant, and is also related to principal component analysis and factor analysis in which it looks for linear combinations of variables which best explain data (McLachlan 2004). LDA has been previously used in the study of diabetes (Cai et al. 2015; Kavakiotis et al. 2017). However, in this work, the following three variants of LDA, which tend to be more robust than the classic one, have been considered: Flexible Discriminant Analysis (FDA) (Hastie, Tibshirani, and Buja 1994); Bagged Flexible Discriminant Analysis (bagFDA), which is bagging wrapper for FDA using multivariate adaptive regression splines; and Sparse Linear Discriminant Analysis (SLDA) (Clemmensen et al. 2011).
Generalized Linear Model (GLM) is a flexible generalization of the ordinary linear regression that allows having target variables with error distribution functions no normally distributed (Nelder and Wedderburn 1972). GLM is able to learn a LASSO or elastic-net model by means of changing the value of its parameter \(\lambda\). LASSO models have previously been used in predicting Diabetic Retinopathy (Oh, Yoo, and Park 2013) and in the diabetic patients’ readmission prediction by means of exploiting health record data (Tutun 2016).
Nearest Shrunken Centroids (NSC) is a classification algorithm that, given a sample, assigns the class of training samples whose centroid is closest to the observation. NCS has been successfully applied in the diagnosis of multiple cancer types from expression profiling (Tibshirani et al. 2002) and in the identification of subsets of genes that best characterize the sample classes (Tibshirani et al. 2003).
Random Forest (RF) is based on the ensemble-based learning paradigm, which constructs several decision trees at training time, and afterward outputs the class that is the mode of all the predicted classes (Ho 1995). RF has been extensively applied in several problems related with diabetes (Gahete et al. 2018), and in the early readmission problem (Duggal et al. 2016b).
In all cases, a 3-times 10-fold cross-validation was conducted for estimating the predictive performance of all the classifiers. The hyperparameters involved in the construction of each classifier were tuned by means of a grid search, assessing a maximum of 10 different values for each possible hyperparameter. On the other hand, three suitable evaluation metrics for classification problems were analyzed to measure the effectiveness of the predictive models, namely the area under ROC curve (AUC), Cohen’s kappa coefficient (CKC) and Matthews correlation coefficient (MCC); these three metrics are less sensitive than other metrics for analyzing unbalanced data. The corresponding versions of these three metrics for multi-class problems were utilized since the studied classification problem is a multi-class one; the target variable has three possible values -“\(<30\)”, “\(>30\)” and “NO”-. The work of Hand and Till (Hand and Till 2001) was followed to compute the AUC in a multi-class scenario, and for our problem, this metric represents the probability that three individuals randomly chosen, each one belonging to a different class of the three possible ones, will be classified in their real classes. To compute the MCC in our multi-class problem, the work of Jurman et al. (Jurman, Riccadonna, and Furlanello 2012) was considered. MCC is able to give a good summary of the confusion matrix yielded by the classification process, allowing to assess how well a classification model is performing independently of the sizes of the classes (Baldi et al. 2000). Regarding CKC, this is a metric that can be directly computed for multi-class problems (Ben-David 2008), and it can measure whether a classifier is performing better than a random prediction.
At the end of the data collection process, the dataset comprised 101,766 encounters described by 50 variables. The dataset is currently available to download at UCI repository, so it is available to other researchers studying the problem of the early readmission of diabetic patients; you can download a zip file that contains two CSV files, the first one named "diabetic_data.csv" that is the dataset with all the variables and without any preprocessing, and the second one named "IDs_mapping.csv" contains the description of each codifier.
Dataset has 49 predictive variables, and a target one (named readmitted) that indicates whether a patient was readmitted within 30 days, after 30 days or no readmitted, \(\{<30, >30, \textrm{NO}\}\). All the predictive variables are numeric or nominal. The following is a brief description of each predictive variable, and their range of values:
Numeric variables:
encounter_id: A unique value for each encounter
patient_number: A unique value for each patient
payer_code: A unique code for each payer
number_diagnoses: Number of diagnoses entered to the system; the existing values are in range \([1, 16]\)
num_emergency: Number of emergency visits of the patient in the year preceding the encounter; values in range \([0, 76]\)
num_inpatient: Inpatient visits of the patient in the year preceding the encounter; values in range \([0, 21]\)
num_medications: Distinct generic names administered during the encounter; values in range \([1, 81]\)
num_label_procedures: Lab tests performed during the encounter; values in range \([1, 132]\)
num_outpatient: Outpatient visits of the patient in the year preceding the encounter; values in range \([0, 42]\)
num_procedures: Other kind of procedures rather than lab tests performed during the encounter; values in range \([0, 6]\)
time_in_hospital: Days between admission and discharge; values in range \([1, 14]\).
Nominal variables:
A1Cresult: Indicates the range of HbA1C test; its possible values are \(\{\)Normal, >=7, >8, None\(\}\)
admission_source: Admission source; 21 possible values \(\{\)Physician referral, Emergency room, Transfer from a hospital...\(\}\)
admission_type: Patient’s admission type; eight possible values \(\{\)Emergency, Urgent, Elective, ...\(\}\)
age: Age of the patient; grouped in 10-year intervals \(\{\)[0-10),[10-20), …, [90-100)\(\}\)
change: Binary variable that indicates if there were any change (dosage or generic name) in diabetes medications; \(\{\)Yes, No\(\}\)
diag_1: Primary diagnosis; 848 possible ICD9 codes
diag_2: Secondary diagnosis; 923 possible ICD9 codes
diag_3: Additional secondary diagnosis; 954 possible ICD9 codes
discharge_diposition: Disposition of the patient at time of discharge; 29 values \(\{\)Discharged to home, Expired, ...\(\}\)
diabetesMed: Binary variable that indicates if there were any administration of diabetes medication; \(\{\)Yes, No\(\}\)
gender: Patient’s gender; \(\{\)Female, Male, Unknown\(\}\)
max_glu_serum: Indicates the range of glucose serum test; \(\{\) Normal, >200, >300, None\(\}\)
medical_specialty: Speciality of the admitting physician; 73 possible values \(\{\)Cardiology, Internal medicine, Emergency Trauma, Nephrology...\(\}\)
race: Patient’s race; five possible values \(\{\)African American, Asian, Caucasian, Hispanic, Other\(\}\)
weight: Patient’s weight; grouped in intervals \(\{\)[25-50), [50-75), [75-100), [100-125),…\(\}\)
Finally, there are 23 variables representing drug dosages: acetohexamide, acarbose, examide, citoglipton chlorpropamide, glimepiride_metformin, glimepiride, glipizide_metformin, glipizide, glyburide glyburide_metformin, insulin, metformin, metformin_pioglitazone, metformin_rosiglitazone, miglitol nateglinide, pioglitazone, repaglinide, rosiglitazone, tolazamide, tolbutamide, and troglitazone; their possible values are \(\{\)Steady, Down, Up, No\(\}\).
This section provides the results of the analysis conducted on the dataset, demonstrating its technical quality. In this section, only a summary of the analysis is presented for easing the reading of the paper; all the results can be consulted on the Web page http://www.uco.es/kdis/early_readmissions/.
The variables encounter_id, patient_number and payer_code were removed since they represent unique identifiers for each record. Those encounters with “Unknown” values in the variable gender were removed. Only three encounters fulfilled this condition, and they also had several missing values in many other variables. Furthermore, these three encounters belonged to non readmitted patients (variable \(readmitted=\)“NO”) and, therefore, this deletion does not affect the group of early readmissions (encounters with variable \(readmitted=\)“\(<30\)”), which is the group of main interest.
As for missing data, they are expected in any medical database since the multiplicity of information sources for data acquisition makes them especially susceptible (Cismondi et al. 2013). The missing values in the dataset are concentrated in the variables weight (96.85% of missing values), medical_specialty (49.08%), race (2.23%), diag_1 (2.06E-4%), diag_2 (3.51E-3%) and diag_3 (1.39E-2%). For the treatment of missing values, the following actions were performed:
The variable weight was considered too sparse and it was not included for further analysis. This high percentage of missing values may be due to the fact that the U.S.A. hospitals and clinics were not required to capture patient’s weight in a structured format prior to the introduction of the HITECH legislation of the American Reinvestment and Recovery Act of 2009.
All missing values in the variables medical_speciality and race were replaced by the value “Missing”.
All missing values in the variables diag_1, diag_2 and diag_3 were replaced by the value “Other”.
After replacing missing values, the actions listed bellow were performed to reduce the levels of some categorical variables.
The levels of variable age was reduced to three levels, “[0-30)”, “[30-60)” and “[60-100)”. In the previous study presented in (Strack et al. 2014), the relationship between the variable age and the readmission probability was analyzed, giving as result that these three intervals have the highest relevance.
The variable discharge_diposition was reduced to two levels, “To_home” and “Otherwise”. As in (Strack et al. 2014), all encounters that resulted in either discharge to a hospice or death were removed (discharge_diposition with Id equal to \(\{\)“11”, “13”, “14”, “19”, “20”, “21”\(\}\)). All discharge_diposition with Id equal to “1” or “8” were assigned to the new level “To_home”, whereas the rest were assigned to the category “Otherwise”.
The levels of the variable admission_source were reduced to three categories, “Emergency_Room”, “Physician/Clinical_Referal” and “Otherwise”. All encounters with Id of admission_source equal to “7” were assigned to the new level “Emergency_Room”. The encounters with Id values equal to “1”-“3” were assigned to the new level “Physician/Clinical_Referal”. On the other hand, the encounters with admission_source equal to “4”-“6”, or “8”-“26” were assigned to the category “Otherwise”.
The levels of the variable medical_speciality were grouped into six more general categories, “Cardiology”, “Family/GeneralPractice”, “InternalMedicine”, “Surgery”, “Other” and “Missing” (as stated before, this last level appeared after replacing the missing values).
The values of the variables diag_1, diag_2 and diag_3 were grouped in nine levels, “Circulatory” (encounters with the ICD9 codes: “390”-“459”and “785”), “Respiratory” (ICD9 codes “460-519” and “786”), “Digestive” (ICD9 codes “520”-“579” and “787”), “Diabetes” (ICD9 codes “250.xx"),”Injury" (ICD9 codes “800”-“999”), “Musculoskeletal” (ICD9 codes “710”-“739”), “Genitourinary” (ICD9 codes “580”-“629”, 788), “Neoplasms” (ICD9 codes “140”-“239”), and “Other” (ICD9 codes “1”-“139”, “240”-“249”, “251”-“279”, “280”-“389”, “630”-“709”, “740”-“759”, “780”-“782”, “784”, “790”-“799” or “E-V”).
Concerning drugs administration, the exploratory data analysis carried out showed that there were several drugs that were never or scarcely prescribed. Those drugs no prescribed at least 1000 times (\(\approx1\%\) of the total of encounters) were removed; to compute the percentage of prescription of a drug, the encounters where the dosage is equal to “Steady”, “Up” or “Down” were summed. Finally, the following drugs were removed: acarbose, acetohexamide, examide, citoglipton, chlorpropamide, glimepiride_metformin, glipizide_metformin, glyburide_metformin, miglitol, metformin_rosiglitazone, metformin_pioglitazone, nateglinide, tolazamide, tolbutamide and troglitazone.
Additionally, a detection of outliers in the numeric variables was performed, by considering the distribution of encounters in the different levels of the variable readmitted. After removing the outliers, the variables num_outpatient and num_emergency had all their values equal to zero and, therefore, these two variables were removed. As final data preprocessing step, all the numeric variables were standardized.
After performing all the aforementioned preprocessing actions, two examples (encounters) became inconsistent and they were removed, i.e. identical examples belonging to different classes. The final preprocesed dataset had 99,337 encounters, described by 29 variables; 28 predictors and the class/target variable -readmitted-. Figure 1 shows that there exists some imbalance between the number of examples in the different classes of encounters, but the imbalance ratio is not so extremely large.
The statistical analysis carried out considering the final 29 predictive variables showed that do no exist a strong correlation between the predictive variables and the target one; in the available Web page many graphs and explanations supporting the results portrayed here can be consulted. Table 1 shows that in the majority of cases the correlation between the predictive variables and the target variable readmitted is very weak, excepting the case of the variable num_inpatient in which the correlation ratio \(\eta\) was moderate. Figure 2 shows that early readmissions had a mean of inpatient visits in the year preceding the encounters significantly higher than those patients readmitted after 30 days or no readmitted.
| readmitted vs. | Correlation | \(p\)-value |
|---|---|---|
| num_diagnoses | 0.15 | 3.1E-24 |
| num_inpatient | 0.24 | 2.5E-30 |
| num_medications | 0.007 | 3.22E-162 |
| num_lab_procedures | 0.003 | 4.71E-56 |
| num_procedures | 0.002 | 3.10E-41 |
| time_in_hospital | 0.006 | 1.53E-127 |
| A1Cresult | 0.021 | 1.11E-17 |
| admission_source | 0.06 | 5.47E-154 |
| admission_type | 0.049 | 5.47E-154 |
| age | 0.03 | 3.84E-42 |
| change | 0.043 | 1.50E-40 |
| diag_1 | 0.05 | 5.31E-98 |
| diag_2 | 0.039 | 3.83E-54 |
| diag_3 | 0.039 | 7.88E-56 |
| discharge_diposition | 0.085 | 3.16E-155 |
| diabetesMed | 0.058 | 4.22E-74 |
| gender | 0.018 | 1.16E-7 |
| max_glu_serum | 0.019 | 1.94E-14 |
| medical_specialty | 0.039 | 1.14E-58 |
| race | 0.04 | 1.24E-58 |
| glimepiride | 0.0 | 1.01E-15 |
| glipizide | 0.014 | 2.74E-7 |
| glyburide | 0.0 | 1.13E-13 |
| insulin | 0.052 | 5.65E-116 |
| metformin | 0.03 | 4.36E-30 |
| pioglitazone | 0.011 | 7.4E-4 |
| repaglinide | 0.016 | 5.96E-10 |
| rosiglitazone | 0.013 | 6.51E-6 |
Table 1. Correlation between the variable readmitted and the 28 predictors.
On the other hand, the existing correlations between pairs of predictive variables were also measured. It was observed a moderate positive correlation between the numeric variables time_in_hospital and num_medications (\(0.415\) with a \(p\)-value\(\textrm{=}1.13\textrm{E-}45\)), and weak positive correlations between time_in_hospital vs. num_lab_procedures (\(0.320\) with a \(p\)-value\(\textrm{=}1.32\textrm{E-}45\)), num_procedures vs. num_medications (\(0.299\) with a \(p\)-value\(\textrm{=}2.12\textrm{E-}34\)), num_medications vs. num_diagnoses (\(0.287\) with a \(p\)-value\(\textrm{=}1.23\textrm{E-}24\)), num_lab_procedures vs. num _medications (\(0.229\) with a \(p\)-value\(\textrm{=}1.126\textrm{E-}45\)) and time_in_hospital vs. num _diagnoses (\(0.228\) with a \(p\)-value\(\textrm{=}2.15\textrm{E-}38\)).
As for pairs of nominal predictive variables, the following strong correlations were detected: insulin vs. change (correlation= \(0.638\) with \(p\)-value\(\textrm{=}2.13\textrm{E-}45\)); admission_type vs. admission_source (correlation=\(0.627\) with \(p\)-value\(\textrm{=}1.98\textrm{E-}34\)); insulin vs. diabetesMed (correlation=\(0.580\) with \(p\)-value\(\textrm{=}1.18\textrm{E-}43\)); and change vs. diabetesMed (correlation=\(0.505\) with \(p\)-value\(\textrm{=}1.35\textrm{E-}64\)). Some of these correlations can be analyzed as logical implications; for example it is expected that encounters with an emergency as the admission type, also had an emergency room as admission source. Figure 3 shows the relationships between the category levels of these nominal variables.
Regarding pairs of numeric and nominal predictive variables, a moderate correlation between num_medications vs. diabetesMed (correlation=\(0.196\) with \(p\)-value\(\textrm{=}2.145\textrm{E-}54\)) was observed. This relationship indicated that encounters with any administration of diabetes medication had a higher number of administered distinct generic names (see Figure 4).
Given the results obtained through this exploratory analysis, it was observed that the number of inpatient visits in the year preceding the encounters (num_inpatient) would be a good estimator to be considered when predicting the early readmissions of diabetic patients. Furthermore, although the rest of predictive variables did not present a strong correlation with the target variable, some existing correlations between predictors could serve to model higher level data relationships for a better prediction of the early readmissions.
Table 2 shows the value of the nine measures that were used for characterizing the complexity of the classification task that represents the studied dataset. It was observed a low F1 value; this does not imply that classes are not linearly separable, but that they cannot be discriminated by hyperplanes parallel to one of the axes of the feature space. F2 value was high, meaning that numeric variables can poorly discriminate the examples of different classes. F3 value was low, which represents the absence of at least one discriminant numeric variable. Furthermore, F4 value was close to 0, meaning a poor collective discriminative power of the numeric variables.
| Measure | Value |
|---|---|
| F1 | 0.375 |
| F2 | 0.924 |
| F3 | 0.204 |
| F4 | 0.213 |
| N1 | 0.648 |
| N2 | 0.997 |
| N3 | 0.489 |
| T1 | 1.00 |
| T2 | 3.55E3 |
Table 2. Complexity measures computed on the studied dataset.
On the other hand, N1 was slightly high, indicating that the majority of the points are close to the class boundary and, therefore, the classification problem has an unclear decision boundary; Luengo et al. (Luengo and Herrera 2012) showed that just having a value of N1 near 0.25 provoked a bad performance of several classifiers. N2, on the other hand, was extremely high, indicating that the examples of the same class are dispersed. Furthermore, the value obtained for N3 indicated that there are small gaps in the decision boundary.
As for the geometry, topology and density of manifolds, T1 reached its maximum value, meaning that a lot of spheres are required to cover the examples of different classes because there are many small clusters of samples. On the other hand, as expected, the T2 value was very high, corroborating that there are much more samples than predictive variables in the dataset.
Through this analysis, it was concluded that the dataset encompasses a challenging classification task and, based on this result, it would be difficult the construction of accurate predictive models.
The performance of 10 classifiers was analyzed in predicting the early readmission of diabetic patients, allowing to measure how complex is the classification problem represented by this dataset; all the graphics showing the results of the parameter tuning process performed for each classifier can be consulted at the available Web page. Table 3 shows the results of the classifiers averaged across all fold executions; the best result attained in each metric (column) is highlighted in bold typeface.
| Classifier | AUC | CKC | MCC |
|---|---|---|---|
| BLR | 0.666 \(\pm\) 0.007 | 0.329 \(\pm\) 0.018 | 0.405 \(\pm\) 0.014 |
| C5.0 | 0.686 \(\pm\) 0.016 | 0.340 \(\pm\) 0.018 | 0.403 \(\pm\) 0.016 |
| CART | 0.684 \(\pm\) 0.065 | 0.337 \(\pm\) 0.146 | 0.406 \(\pm\) 0.017 |
| CTree | 0.694 \(\pm\) 0.004 | 0.326 \(\pm\) 0.008 | 0.399 \(\pm\) 0.009 |
| FDA | 0.686 \(\pm\) 0.005 | 0.327 \(\pm\) 0.008 | 0.404 \(\pm\) 0.007 |
| bagFDA | 0.689 \(\pm\) 0.004 | 0.327 \(\pm\) 0.006 | 0.404 \(\pm\) 0.005 |
| SLDA | 0.683 \(\pm\) 0.003 | 0.299 \(\pm\) 0.026 | 0.364 \(\pm\) 0.033 |
| GLM | 0.683 \(\pm\) 0.004 | 0.297 \(\pm\) 0.007 | 0.364 \(\pm\) 0.007 |
| NSC | 0.666 \(\pm\) 0.004 | 0.294 \(\pm\) 0.007 | 0.364 \(\pm\) 0.006 |
| RF | 0.708 \(\pm\) 0.004 | 0.345 \(\pm\) 0.009 | 0.395 \(\pm\) 0.012 |
Table 3. Average predictive performance of the classifiers.
The classifiers RF, CTree and bagFDA obtained the best results on the AUC metric. Regarding metric CKC, the classifiers RF and C5.0 attained the best performance, whilst the classifiers CART, BLR, bagFDA and FDA reached the best performance on metric MCC. It is noteworthy that RF obtained the best results for the metrics AUC and CKC, and it reached a slightly lower value than the best result obtained in metric MMC (the difference with respect to the best value was 0.01). As for the tree-based classifiers (C5.0, CART, CTree and RF), RF was the algorithm that obtained the best average ranking, whereas C5.0 was the second one; C5.0 obtained the third position in the metric AUC, and the second position in the metrics CKC and MCC. Regarding the three discriminant analysis-based classifiers (FDA, bagFDA and SLDA), the best classifier was bagFDA and the worst one SLDA. Finally, if we consider the 10 classifiers, and the three evaluation metrics, the two classifiers with the worst performance levels were GLM and NSC.
Analyzing the prediction levels, the maximum AUC value attained was 0.708 (AUC ranges from 0.5 to 1.0), for CKC was 0.345, whereas a value of 0.406 was obtained for MCC; CKC and MCC range from -1 to 1. The performance levels obtained by all classifiers confirmed that the dataset presents a challenging classification problem, confirming the data complexity analysis presented in the previous section. Taking into account that diabetes is a disease that consumes a large proportion of healthcare resources, these predictive levels may be not enough for the implementation of high reliable strategies that allow hospital systems identify those patients with risk of being readmitted within a short period of time.
All classifiers used in the experimental study have a built-in variable selection process that allows to analyze and determine the most relevant set of variables for predicting early readmissions. Figures 5-7 show the importance of variables, which were averaged across all fold executions, and finally they were average by group of classifiers. Discriminant analysis-based classifiers agreed that the three most relevant predictive variables were num_inpatient, num_diagnoses, and num_medications, followed by the variable discharge_disposition which had a lower relevance; the encounters discharged to home provided more relevant information for the correct classification of all encounters. Regarding the tree-based classifiers, they estimated that the variables most relevant for predicting the early readmissions were num_inpatient, num_medications, num_diagnoses and time_in_hospital, in this order. On the other hand, the classifiers BLR, GLM and NSC also estimated that the most relevant variables were the same that those ones detected by the tree-based classifiers, but they also included discharge_disposition.
Based on this analysis, it was observed a tendency to consider the numeric variables as those ones with the best predictive capability, whereas the nominal variables did not have relevant contributions in discriminating the encounters. This result was expected, since the conducted analysis of statistical relationships showed that there exist very weak associations between the nominal predictive variables and the target variable readmitted, whereas there was a quite moderate correlation between the variable num_inpatient and readmitted; it is noteworthy that all classifiers coincided that num_inpatient is the most relevant variable. This result agreed with other works that have previously exposed that the early readmission of diabetic patients is significantly associated with the quality of inpatient care (Ashton et al. 1995).
On the other hand, the variables num_diagnoses, num_medications, and time_in_hospital were also relevant in constructing the classifiers. Albeit these three variables had weak individual correlations with the target one, it seems that they allow to model stronger relationships in data if they were analyzed jointly; for example, it was previously observed a moderate correlation between the variables time_in_hospital and num_medications).
The statistical analysis conducted on the dataset revealed that the correlations between the predictive variables and the class variable are weak, excepting the strong detected relation indicating that an early readmitted patient commonly have a higher number of inpatient visits in the year preceding the encounter.
On the other hand, nine data complexity measures demonstrated that the studied dataset entails a classification problem very difficult and complex to solve. Later, this observation was confirmed by means of assessing 10 classifications algorithms on the dataset. All the predictive models agreed with the conducted data statistical analysis, confirming that the number of inpatient visits in the year preceding the encounter is the most relevant variable for predicting the early readmission of a diabetic patient.
The studied dataset can be used as a benchmark either for testing new classification algorithms or performing other knowledge discovery tasks, such as the extraction of general rules with pattern mining methods, or the discovery of subgroups of encounters describing subsets of the population that are sufficiently large and statistically unusual. Future work can also study the application of deep learning models in predicting the early readmissions of diabetic patients; this type of learning models have demonstrated to be highly effective in the solution of a large variety of complex predictive problems mainly due to they do not require to conduct expensive feature engineering process, and they also have a great capability to model high order data relationships. Finally, it is noteworthy that the studied dataset is suitable to be used in data complexity analysis, a field where researchers often lack high dimensional real-world datasets.
Data analysis were performed mainly using R language for statistical computing v3.4.3 (R Core Team 2018), specifically CARET R-package v6.0-78 (Kuhn 2008) was employed for preprocessing data and executing all the classifiers. Also, DCoL library (Orriols-Puig, Macia, and Ho 2010) was used to calculate all the metrics that characterize the complexity of the classification task; this is a C++ library and it was compiled with g++ v4.0. All the code is accessible via GitHub repository at https://github.com/kdis-lab/Early_readmission_diabetic_patients.
This research was supported by the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund, projects TIN2017-83445-P. Also, we want to thank the early help of Juan Luis Olmo, Beata Strack, Jonathan P. DeShazo and John N. Clore.
Supplementary Information is available for this work at http://www.uco.es/kdis/early_readmissions/.
Ashton, C. M., D. H. Kuikendall, M. L. Johnson, N. P. Wray, and L.Wu. 1995. “The Association Between the Quality of Inpatient Care and Early Readmission.” Annals of Internal Medicine 122 (6): 415–21.
Baldi, P., S. Brunak, Y. Chauvin, C. Andersen, and H. Nielsen. 2000. “Assessing the Accuracy of Prediction Algorithms for Classification: An Overview.” Bioinformatics 16: 412–24.
Ben-David, A. 2008. “About the relationship between ROC curves and Cohen’s kappa.” Engineering Applications of Artificial Intelligence 21 (6): 874–82.
Breiman, L., J. Friedman, C. J. Stone, and R. A. Olshen. 1984. “Classification and Regression Trees.” CRC Press.
Cai, L., H. Wu, D. Li, K. Zhou, and F. Zou. 2015. “Type 2 Diabetes Biomarkers of Human Gut Microbiota Selected via Iterative Sure Independent Screening Method.” PLoS One 10 (10): e0140827.
Cismondi, F., A. Fialho, S. Vieira, S. Reti, and J. Sousa. 2013. “Missing Data in Medical Databases: Impute, Delete or Classify?” Artificial Intelligence in Medicine 58 (1): 63–72.
Clemmensen, L., T. Hastie, D. Witten, and B. Ersboll. 2011. “Sparse Discriminant Analysis.” Technometrics 53 (4): 406–13.
Duggal, R., S. Shukla, S. Chandra, B. Shukla, and S. K. Khatri. 2016a. “Impact of selected pre-processing techniques on prediction of risk of early readmission for diabetic patients in India.” International Journal of Diabetes in Developing Countries 36 (4): 469–76.
———. 2016b. “Predictive Risk Modelling for Early Hospital Readmission of Patients with Diabetes in India.” International Journal of Diabetes in Developing Countries 36 (4): 519–28.
Dungan, K. M. 2012. “The Effect of Diabetes on Hospital Readmissions.” Journal of Diabetes Science and Technology 6 (5): 1045–52.
Eby, E., C. Hardwick, M. Yu, S. Gelwicks, K. Deschamps, J. Xie, and T. George. 2014. “Predictors of 30 Day Hospital Readmission in Patients with Type 2 Diabetes: A Retrospective, Case-Control, Database Study.” Current Medical Research & Opinion 31 (1): 107–14.
Freund, Y., and R. E. Schapire. 1997. “A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting.” Journal of Computer and System Sciences 55 (1): 119–39.
Friedman, J., T. Hastie, and R. Tibshirani. 2000. “Additive Logistic Regression: A Statistical View of Boosting.” Annals of Statistics 28 (2): 337–407.
Futoma, J., J. Morris, and J. Lucas. 2015. “A Comparison of Models for Predicting Early Hospital Readmissions.” Journal of Biomedical Informatics 56: 229–38.
Gahete, M. D., M. del Rio-Moreno, A. Camargo, J. F. Alcala-Diaz, E. Alors-Perez, J. Delgado-Lista, O. Reyes, et al. 2018. “Changes in Splicing Machinery Components Influence, Precede, and Early Predict the Development of Type 2 Diabetes: From the Cordioprev Study.” EBioMedicine 37: 356–65.
Garrison, M. M., W. J. Katon, and L. P. Richardson. 2005. “The Impact of Psychiatric Comorbidities on Readmissions for Diabetes in Youth.” Diabetes Care 28 (9): 2150–4.
Golmohammadi, D., and N. Radnia. 2016. “Prediction Modeling and Pattern Recognition for Patient Readmission.” International Journal of Production Economics 171: 151–61.
Hand, D. J., and R. J. Till. 2001. “A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems.” Machine Learning 45 (2): 171–86.
Hastie, T., R. Tibshirani, and A. Buja. 1994. “Flexible Discriminant Analysis by Optimal Scoring.” Journal of the American Statistical Association 89 (428): 1255–70.
Healy, S. J., D. Black, C. Harris, A. Lorenz, and K. M. Dungan. 2013. “Inpatient Diabetes Education Is Associated with Less Frequent Hospital Readmission Among Patients with Poor Glycemic Control.” Diabetes Care 36 (10): 2960–7.
Ho, T. K. 1995. “Random Decision Forests.” Montreal, Canada.
Ho, T. K., and M. Basu. 2002. “Complexity Measures of Supervised Classification Problems.” IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (3): 289–300.
Hothorn, T., K. Hornik, and A. Zeileis. 2006. “Unbiased Recursive Partitioning: A Conditional Inference Framework.” Journal of Computational and Graphical Statistics 15 (3): 651–74.
Jiang, H. J., D. Stryer, B. Friedman, and R. Andrews. 2003. “Multiple Hospitalizations for Patients with Diabetes.” Diabetes Care 26 (5): 1421–6.
Jurman, G., S. Riccadonna, and C. Furlanello. 2012. “A Comparison of MCC and CEN Error Measures in Multi-Class Prediction.” PLoS One 7 (8): e41882.
Kavakiotis, I., O. Tsave, A. Salifoglou, N. Maglaveras, I. Vlahavas, and I. Chouvarda. 2017. “Machine Learning and Data Mining Methods in Diabetes Research.” Computational and Structural Biotechnology Journal 15: 104–16.
Kuhn, M. 2008. “Caret Package.” Journal of Statistical Software 28 (5): 1–26.
Leppin, A. L., M. R. Gionfriddo, M. Kessler, J. P. Brito, F. S. Mair, K. Gallacher, and H. H. Ting. 2014. “Preventing 30-Day Hospital Readmissions: A Systematic Review and Meta-Analysis of Randomized Trials.” JAMA Internal Medicine 174 (7): 1095–1107.
Luengo, J., and F. Herrera. 2012. “Shared Domains of Competence of Approximate Learning Models Using Measures of Separability of Classes.” Information Sciences 185 (1): 43–65.
Mani, S., Y. Chen, T. Elasy, W. Clayton, and J. Denny. 2012. “Type 2 diabetes risk forecasting from EMR data using machine learning.”
McLachlan, G. 2004. Discriminant Analysis and Statistical Pattern Recognition. Vol. 544. John Wiley & Sons.
Nelder, J., and R. Wedderburn. 1972. “Generalized Linear Models.” Journal of the Royal Statistical Society. Series A (General). Blackwell Publishing 135 (3): 370–84.
Oh, E., T. K. Yoo, and E. C. Park. 2013. “Diabetic Retinopathy Risk Prediction for Fundus Examination Using Sparse Learning: A Cross-Sectional Study.” BMC Medical Informatics and Decision Making 13 (1): 106.
Orriols-Puig, A., N. Macia, and T. K. Ho. 2010. “Documentation for the data complexity library in C++.” 2009001. Universitat Ramon Llull, La Salle.
Quinlan, J. R. 1993. “C4. 5: Programming for Machine Learning.” Morgan Kauffmann 38.
R Core Team. 2018. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Rubin, D. J. 2015. “Hospital Readmission of Patients with Diabetes.” Current Diabetes Reports 15 (4): 17.
Rubin, D. J., S. H. Golden, M. E. McDonnell, and H. Zhao. 2017. “Predicting Readmission Risk of Patients with Diabetes Hospitalized for Cardiovascular Disease: A Retrospective Cohort Study.” Journal of Diabetes and Its Complications 31 (8): 1332–9.
Rubin, D. J., E. A. Handorf, S. H. Golden, D. B. Nelson, M. E. McDonnell, and H. Zhao. 2016. “Development and Validation of a Novel Tool to Predict Hospital Readmission Risk Among Patients with Diabetes.” Endocrine Practice 22 (10): 1204–15.
Strack, B., J. P. DeShazo, C. Gennings, J. L. Olmo, S. Ventura, K. J. Cios, and J. N. Clore. 2014. “Impact of HbA1c Measurement on Hospital Readmission Rates: Analysis of 70,000 Clinical Database Patient Record.” BioMed Research International, 1–11.
Tibshirani, R., T. Hastie, B. Narasimhan, and G. Chu. 2002. “Diagnosis of Multiple Cancer Types by Shrunken Centroids of Gene Expression.” 6567-6572.
———. 2003. “Class prediction by nearest shrunken centroids, with applications to DNA microarrays.” Statistical Science, 104–17.
Tutun, S. 2016. “A Meta-heuristic LASSO Model for Diabetic Readmission Prediction.”
Verhaegh, K. J., J. L. MacNeil-Vroomen, S. Eslami, S. E. Geerlings, S. E. de Rooij, and B. M. Buurman. 2014. “Transitional Care Interventions Prevent Hospital Readmissions for Adults with Chronic Illnesses.” Health Affairs 33 (9): 1531–9.
Wei, N. J., D. Wexler, D. M. Nathan, and R. W. Grant. 2013. “Intensification of Diabetes Medication and Risk for 30-Day Readmission.” Diabetic Medicine: A Journal of the British Diabetic Association 30 (2): e56–e62.
Y., In-Kwon, and J. A. Richard. 2000. “A New Family of Power Transformations to Improve Normality or Symmetry.” Biometrika 87 (4): 954–59.