How is the success of a data mining model measured?
The quality of data mining models of this type is measured along two dimensions. Say one million transactions are scored as described above and there are 1000 transactions with a score about 900 that are examined in detail. Say that these 1000 transactions contain 100 bad ones. Assume also that the one million transactions contain 200 bad ones. Then 100/200 or 50% of the bad ones are detected, so we say that the detection rate is 50%. In addition, for each bad one found, 10 other good transactions are examined, so we say that the false positive rate is 10 to 1. Notice that the incidence of fraudulent transactions is 200/1,000,000 or .0002 = .02%. To catch more bad guys requires more analysts and bothering more good guys. If more analysts were available, then one might lower the threshold of transactions to examine from 900 to 850.
