


CORRESPONDENCE 

Year : 2017  Volume
: 145
 Issue : 4  Page : 571572 

Choice of criterion used in the receiver operating characteristic analysis
Farrokh Habibzadeh^{1}, Parham Habibzadeh^{2}, Mahboobeh Yadollahie^{3}
^{1} Shiraz University of Medical Sciences; Petroleum Industry Health Organization, Shiraz, Iran ^{2} Shiraz University of Medical Sciences, Shiraz, Iran ^{3} Petroleum Industry Health Organization, Shiraz, Iran
Date of Submission  16Dec2016 
Date of Web Publication  28Aug2017 
Correspondence Address: Farrokh Habibzadeh Shiraz University of Medical Sciences; Petroleum Industry Health Organization, Shiraz Iran
Source of Support: None, Conflict of Interest: None  Check 
DOI: 10.4103/ijmr.IJMR_2002_16
How to cite this article: Habibzadeh F, Habibzadeh P, Yadollahie M. Choice of criterion used in the receiver operating characteristic analysis. Indian J Med Res 2017;145:5712 
How to cite this URL: Habibzadeh F, Habibzadeh P, Yadollahie M. Choice of criterion used in the receiver operating characteristic analysis. Indian J Med Res [serial online] 2017 [cited 2020 Jul 12];145:5712. Available from: http://www.ijmr.org.in/text.asp?2017/145/4/571/213727 
Sir,
We read with interest the article of Mohan et al^{[1]} on determination of glycosylated haemoglobin (HbA_{1c}) cutoff value for the diagnosis of type 2 diabetes in Rayalaseema area in Andhra Pradesh, Southern India^{[1]}. They mention that various studies have reported different HbA_{1c} cutoff values from ≥5.6 per cent (38 mmol/mol) to ≥7.0 per cent (53 mmol/mol) and that 'age, ethnicity, genetic makeup, erythrocyte lifespan and erythrocyte environment' are the most likely causes of the observed variance. We believe that in selecting the criterion used to derive the most appropriate test cutoff value in the receiver operating characteristic (ROC) curve analysis, we should also consider other important contributing factors^{[2]}.
Mohan et al^{[1]} mentioned that based on the article of Perkins and Schisterman^{[3]}, they used the cutoff value corresponding to the point on the ROC curve with the minimum distance from the upper left corner of the unit square, where both the test sensitivity (Se) and specificity (Sp) were equal to 1. However, Perkins and Schisterman^{[3]} clearly recommended the use of Youden's index (Se + Sp − 1) and warned about using the minimum distance criterion.
It has been shown that neither the minimum distance nor maximizing the Youden's index, though commonly used by researchers, is the best criterion for the calculation of the most appropriate test cutoff values^{[2],[4]}. To reach the most appropriate value, it is imperative to also take into account the 'pretest probability' (pr) of the disease as well as the relative cost of a false negative test result as compared to a false positive result (C). Generally, in areas where the pr (disease prevalence, where there is no other information about the person tested) and the C are higher, to obtain better results, we should set the test cutoff point at a lower value^{[2],[4]}. It can be shown that maximizing the weighted Number Needed to Misdiagnose (wNNM)^{[5]}, defined as follows, is corresponding to the most appropriate test cutoff value:
To calculate the wNNM, we need to know estimations of the pr and C and also the Se and Sp corresponding to each point on the ROC curve. From [Table 2] of Mohan et al's^{[1]} article, we know that of the 342 persons examined in the training data set, there were 139 newly diagnosed patients with type 2 diabetes, translating into a pr of 0.41. To determine the Se and Sp corresponding to each point on the ROC curve, we retrieve parts of the data from [Figure 2]A and [Figure 3] (inset graph)^{[1]} by a digitizer (CalComp, USA).
If we miss a patient with type 2 diabetes (a false negative test result), the patient will return and present with a more advanced stage of the disease with more complications needing more care. On the other hand, labelling a person to have diabetes, while he or she really does not (a false positive test result), causes him/her to be further tested, plus the time and the psychological trauma to them and their family members. Even after a normal test, it is difficult to believe that he or she really does not have the disease. Assume the cost of making a false negative test result is five times the cost of a false positive result (C=5). Considering a pr of 0.41 and C of 5, the maximum wNNM of 3.74 occurs at a point on ROC curve with a Se of 0.91, Sp of 0.85, a positive predictive value (PPV) of 0.81, and a negative predictive value (NPV) of 0.93, corresponding to a cutoff value of >6.3 per cent (45 mmol/mol). This value is accidentally equal to the value derived through using the minimum distance method used by the authors^{[1]} ([Figure 1], the white circle). The Youden's index at this very point has also the maximum value of 0.76. However, all the assumptions made by the authors would not necessarily be correct, even for their region.  Figure :1 Part of the receiver operating characteristic curve plotted based on the data retrieved from [Figure 2]a of Ref. 1. The white circle corresponds to the cutoff point derived by the authors of the article. The green square corresponds to a higher cutoff value when the prior probability is considered 9 per cent (the prevalence of type 2 diabetes in general population) instead of 41 per cent (the prevalence in the referral centre where the study was conducted), and the red triangle corresponds to a lower cutoff value, when the cost of a false negative test result is assumed 10 (instead of 5) times a false positive result.
Click here to view 
Suppose we want to use HbA_{1c} as a screening test in general population in the studied region. Under such circumstance, the pr is no longer 0.41; it is 0.09  the prevalence of type 2 diabetes in the studied region, Rayalaseema area in Andhra Pradesh, Southern India^{[6]}. Considering a pr (here, the prevalence) of 0.09 and a C of 5 (as before), the maximum wNNM corresponds to a Se of 0.80, Sp of 0.93, PPV of 0.52 and NPV of 0.98, corresponding to a cutoff value of >6.8 per cent (51 mmol/mol) ([Figure 1], the green square). The centre where the study was conducted was probably a referral centre; all persons studied were 'suspected to have type 2 diabetes', probably based on their signs or symptoms (polyuria, polydipsia, etc.), hence, a higher pr of 0.41. As one can see, where the pr of a disease is higher, we need to use a lower test cutoff value to obtain the best result^{[2]}. Therefore, it is very important to bear in mind to use an HbA_{1c} cutoff of >6.3 per cent when practicing in that referral centre, and a value of >6.8 per cent when practicing in a general clinic.
Now, suppose one in the said referral centre (pr=0.41) believes that the relative cost of a false negative test result is 10 times the cost of a false positive result (C=10, not 5). Then, the maximum wNNM corresponds to a Se of 0.96, Sp of 0.65, PPV of 0.66, and NPV of 0.96, corresponding to a cutoff value of >5.9 per cent (41 mmol/mol) ([Figure 1], the red triangle). Note that in the calculation of the cost, we should not only consider financial costs incurred but also take into account all other costs (psychological, legal, etc.). These calculations are mainly based on the results of costeffectiveness and benefitrisk studies. Apparently, the higher the relative cost of a false negative to a false positive test results, the lower test cutoff value should be set^{[2]}.
Many of the criteria nowadays commonly used to determine the most appropriate test cutoff value, consider neither the pr of the disease of interest nor the costs of false test results. It can however be shown that these criteria (e.g., maximizing the Youden's index) are special cases of the maximizing wNNM^{[2]}. To obtain better results, we, therefore, suggest using the method of maximizing wNNM, which can easily be calculated based on the output data of most statistical software programmes.
Conflicts of Interest: None.
References   
1.  Mohan A, Reddy SA, Sachan A, Sarma KVS, Kumar DP, Panchagnula MV, et al. Derivation & validation of glycosylated haemoglobin (HbA _{1c}) cutoff value as a diagnostic test for type 2 diabetes in south Indian population. Indian J Med Res 2016; 144 : 2208. 
2.  Habibzadeh F, Habibzadeh P, Yadollahie M. On determining the most appropriate test cutoff value: the case of tests with continuous results. Biochem Med ( Zagreb) 2016; 26 : 297307. 
3.  Perkins NJ, Schisterman EF. The inconsistency of “optimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol 2006; 163 : 6705. 
4.  Sox HC, Higgins MC, Owens DK. Medical decision making, 2 ^{nd} ed. Oxford, UK: WileyBlackwell; 2013. 
5.  Habibzadeh F, Yadollahie M. Number needed to misdiagnose: A measure of diagnostic test effectiveness. Epidemiology 2013; 24 : 170. 
6.  Reddy NK, Kumar DN, Rayudu NV, Sastry BK, Raju BS. Prevalence of risk factors for coronary atherosclerosis in a crosssectional population of Andhra Pradesh. Indian Heart J 2002; 54 : 697701. 
[Figure 1]
