Data Drift Detection for Image Classifiers – Data Science Blog by Domino


This text covers easy methods to detect knowledge drift for fashions that ingest picture knowledge as their enter to be able to stop their silent degradation in manufacturing. Run the instance in a complementary Domino project.

In the actual phrase, knowledge is recorded by completely different programs and is consistently altering. How knowledge is collected, handled and remodeled impacts the info. Change could happen with the introduction of noise as a consequence of mechanical put on and tear to bodily programs or if a basic shift within the underlying generative course of happens (e.g., a change within the rate of interest by a regulator, a pure calamity, the introduction of a brand new competitor out there or a change in a enterprise technique/course of, and so forth.). Such adjustments have ramifications for the accuracy of predictions and necessitate the necessity to examine that the assumptions made through the growth of fashions nonetheless maintain good when fashions are in manufacturing.

Within the context of machine studying, we think about knowledge drift1 to be the change in mannequin enter knowledge that results in a degradation of mannequin efficiency. Within the the rest of this text, we will cowl easy methods to detect knowledge drift for fashions that ingest picture knowledge as their enter to be able to stop their silent degradation in manufacturing.

Given the proliferation of curiosity in deep studying within the enterprise, fashions that ingest non conventional types of knowledge resembling unstructured textual content and pictures into manufacturing are on the rise. In such instances, strategies from statistical course of management and operations analysis that rely totally on numerical knowledge are onerous to undertake and necessitates a brand new strategy to monitoring fashions in manufacturing. This text explores an strategy that can be utilized to detect knowledge drift for fashions that classify/rating picture knowledge.

Our strategy doesn’t make any assumptions concerning the mannequin that has been deployed, but it surely requires entry to the coaching knowledge used to construct the mannequin and the prediction knowledge used for scoring. The intuitive strategy to detect knowledge drift for picture knowledge is to construct a machine discovered illustration of the coaching dataset and to make use of this illustration to reconstruct knowledge that’s being offered to the mannequin. If the reconstruction error is excessive then the info being offered to the mannequin is completely different from what it was educated on. The sequence of operations is as follows:

  1. Study a low dimensional illustration of the coaching dataset (encoder)
  2. Reconstruct a validation dataset utilizing the illustration from Step 1 (decoder) and retailer reconstruction loss as baseline reconstruction loss
  3. Reconstruct batch of knowledge that’s being despatched for predictions utilizing encoder and decoder in Steps 1 and a couple of; retailer reconstruction loss

If reconstruction lack of dataset getting used for predictions exceeds the baseline reconstruction loss by a predefined threshold set off an alert.

The steps beneath present the related code snippets that cowl the strategy that has been detailed above. The entire code of the undertaking is accessible and could be forked from the Image_Drift_Detection project on

The pocket book (Convolutional_AutoEncoder.ipynb) that’s obtainable at Image_Drift_Detection project

is laid out as follows:

  1. Practice a convolutional autoencoder on the MNIST dataset2; use the validation loss because the baseline to check drift for brand spanking new datasets.
  2. Add noise to the MNIST dataset and try to reconstruct the noisy MNIST dataset; observe reconstruction loss. (for use instead baseline if required).
  3. Reconstruct the nonMNIST3 dataset utilizing the convolutional autoencoder in-built Step 1 and examine the reconstruction loss with the validation lack of the MNIST dataset or the reconstruction loss on the noisy MNIST knowledge set.

We now dive into the related code snippets to detect drift on picture knowledge.

Step 1: Set up the required dependencies for the undertaking by including the next to your Dockerfile. You may as well use the Image_Drift_Keras_TF setting in because it has all of the libraries/dependencies preinstalled.

RUN pip set up numpy==1.13.1
RUN pip set up tensorflow==1.2.1
RUN pip set up Keras==2.0.6

Step 2: Begin a Jupyter pocket book workspace and cargo the required libraries.

#Load libraries
from keras.layers import Enter, Dense, Conv2D, MaxPool2D, UpSampling2D, MaxPooling2D
from keras.fashions import Mannequin
from keras import backend as Okay
from keras import regularizers
from keras.callbacks import ModelCheckpoint

from keras.datasets import mnist
import numpy as np
import matplotlib.pyplot as plt

Step 3: Specify the structure for the autoencoder.

#Specify the structure for the auto encoder
input_img = Enter(form=(28, 28, 1))

# Encoder
x = Conv2D(32, (3, 3), activation='relu', padding='similar')(input_img)
x = MaxPooling2D((2, 2), padding='similar')(x)
x = Conv2D(32, (3, 3), activation='relu', padding='similar')(x)
encoded = MaxPooling2D((2, 2), padding='similar')(x)

# Decoder
x = Conv2D(32, (3, 3), activation='relu', padding='similar')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(32, (3, 3), activation='relu', padding='similar')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='similar')(x)

autoencoder = Mannequin(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

Step 4: Generate the take a look at, prepare and noisy MNIST knowledge units.

# Generate the prepare and take a look at units
(x_train, _), (x_test, _) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = np.reshape(x_train, (len(x_train), 28, 28, 1))
x_test = np.reshape(x_test, (len(x_test), 28, 28, 1))

# Generate some noisy knowledge
noise_factor = 0.5
x_train_noisy = x_train + noise_factor * np.random.regular(loc=0.0, scale=1.0, measurement=x_train.form)
x_test_noisy = x_test + noise_factor * np.random.regular(loc=0.0, scale=1.0, measurement=x_test.form)

x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)

Step 5: Practice the convolutional autoencoder.

# Checkpoint the mannequin to retrieve it later
cp = ModelCheckpoint(filepath="autoencoder_img_drift.h5",

# Retailer coaching historical past
historical past = autoencoder.match(x_train_noisy, x_train,
                validation_data=(x_test_noisy, x_test),
                callbacks = [cp, plot_losses]).historical past

The validation loss we obtained is 0.1011.

Step 6: Get the reconstruction loss for the noisy MNIST knowledge set.

x_test_noisy_loss = autoencoder.consider(x_test_noisy,decoded_imgs)

Pattern of noisy enter knowledge (high row) and reconstructed pictures (backside row)

The reconstruction error that we obtained on the noisy MNIST knowledge set is 0.1024. In contrast with the baseline reconstruction error on the validation dataset it is a 1.3% improve within the error.

Step 7: Get the reconstruction loss for the nonMNIST dataset; examine with validation loss for MNIST dataset and noisy MNIST.

non_mnist_data_loss = autoencoder.consider(non_mnist_data,non_mnist_pred)

Pattern of nonMNIST knowledge (high row) and its corresponding reconstruction (backside row)

The reconstruction error that we obtained on the nonMNIST knowledge set is 0.1458. In contrast with the baseline reconstruction error on the validation dataset, it is a 44.2% improve within the error.

From this instance, it’s clear that there’s a spike within the reconstruction loss when the convolution encoder is made to reconstruct a dataset that’s completely different than the one used to coach the mannequin. The subsequent step on account of this detection is to both retrain the mannequin with new knowledge or to research what led to a change within the knowledge.

On this instance, we noticed {that a} convolutional autoencoder is able to quantifying variations in picture datasets on the idea of a reconstruction error. Utilizing this strategy of evaluating reconstruction errors, we are able to detect adjustments within the enter that’s being offered to picture classifiers. Nonetheless, enhancements could be made to this strategy sooner or later. For one, whereas we used a trial-and-error strategy to choose the structure of the convolutional autoencoder, by utilizing a neural structure search algorithm we are able to receive a way more correct autoencoder mannequin. One other enchancment could possibly be utilizing a knowledge augmentation technique on the coaching dataset to make the autoencoder extra sturdy to noisy knowledge and variances in rotations and translations of pictures.

If you need to know extra about how you need to use Domino to watch your manufacturing fashions, you possibly can attend our Webinar on “Monitoring Models at Scale”. It’ll cowl steady monitoring for knowledge drift and mannequin high quality.


  1. Gama, Joao; Zliobait, Indre; Bifet, Albert; Pechenizkiy, Mykola; and Bouchachia, Abdelhamid. “A Survey on Idea Drift Adaptation” ACM Computing Survey Quantity 1, Article 1 (January 2013)
  2. LeCun, Yann; Corinna Cortes; Christopher J.C. Burges. “The MNIST Database of handwritten digits“.
  3. notMNIST dataset


Source link

Write a comment