post Details

Why data science and machine learning are important for your business?

In this digital era, data-driven decisions are more important than ever for your business. When you make data-driven decisions, not only you can be sure in your decisions, but also you are ahead of the competition who is not making data-driven decisions. There are many interesting phenomena and topics in data science, but in this blog post, we are going to describe the problem of learning from imbalanced data, and how we at Quant Coding deal with it.

The problem with imbalanced data

The problem which arises when you try to train a classifier on a dataset where the classes are not equally represented is usually referred to as the imbalanced data problem. For example, in a binary classification setting, 99% of the examples may be from one class and only 1% of the examples may be from the other class. And usually, the minority class is the one that we are interested in.

Examples of problems with natural imbalanced data are:

• credit card fraud detection, from all the logged transaction only a small fraction are malicious
• cancer patients, out of all the patients who go for diagnostics, only a small fraction have cancer
• flight delays, most of the flights are on time, with only a small fraction severely delayed

How we tackle imbalanced data at Quant Coding:

1. Under-sample the majority class – One way to get a balanced dataset out of an imbalanced is to under-sample the majority class. This simply means randomly dropping examples from the majority class. The drawback is that we are losing data and that generally in machine learning is a bad thing.
2. Generate examples from the minority class – Another way is to model the distribution of the features of the minority class, and then generate artificial examples from the minority class. The drawback of this approach is that generating a lot of examples, may confuse the model and lead to bad performance.
3. Suggest the model that we are dealing with unbalanced data – Models usually have a class weights attribute that tells the model which class is the minority, and which is the majority class, and by how much is the majority class overrepresented over the minority class. This enables the model to focus more on the minority class while training.
4. Use a model that is aware of the unbalanced data – These are models that are developed for training on imbalanced data. The best of them in our experience is the Balanced Random Forest. In summary, it is based on the Random Forst model and it is an ensemble learning method. When training the constituting decision trees, the Balanced Random Forst, for every decision tree does a different under-sampling of the majority class. This leads to a better model overall.

Evaluation of imbalanced data:

Evaluating models trained on imbalanced data is different from evaluating other models.

1. Distribution of the imbalanced testing set – since the testing set should be representative of the real-world data on which the model will be used, should also be imbalanced. For example, when detecting credit-card frauds in production, most of the transactions will be not malicious, and that should also be the case with the training set. This way we are sure that we are evaluating the model in a setting as close to production as possible, reducing the surprises we may encounter in production.
2. Metrics – Using accuracy may be misleading. For example, a classifier that labels all the examples with the same label as the majority class may get an accuracy of 99%, and such a classifier is no use for us. Therefore, we need to be careful which metrics we will use to evaluate the models. The best metrics to use are:

1. Recall – Recall is defined as how many of the real positive examples, our models classify as positive. In the case of the cancer patients, we may be interested in models with better recall, since we don’t want to miss a patient who has cancer, and label him as without cancer.
2. Precision – Precision is defined as how many of the examples labeled as positive by our classifier, are actually positive. In the case of credit card fraud transactions, we may want a higher precision, in order not to block customer’s credit cards when there is no malicious use.
3. F1 score – The F1 scores capture both the precision and recall in one metric. Actually, it is a harmonic mean of the precision and the recall of the model.
4. Confusion matrix – Looking confusion matrix we can get a feeling of how our classifier performs. Furthermore, all these metrics can be computed directly from the confusion matrix.