.. _searchimageskerasrst: ======================================== Search images with deep learning (keras) ======================================== .. only:: html **Links:** :download:`notebook `, :downloadlink:`html `, :download:`PDF `, :download:`python `, :downloadlink:`slides `, :githublink:`GitHub|_doc/notebooks/explore/search_images_keras.ipynb|*` Images are usually very different if we compare them at pixel level but that’s quite different if we look at them after they were processed by a deep learning model. We convert each image into a feature vector extracted from an intermediate layer of the network. .. code:: ipython3 from jyquickhelper import add_notebook_menu add_notebook_menu() .. contents:: :local: .. code:: ipython3 %matplotlib inline Get a pre-trained model ----------------------- We choose the model described in paper `MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications `__. Pre-trained models are available at `deep-learning-models/releases `__. .. code:: ipython3 from keras.applications.mobilenet import MobileNet model = MobileNet(input_shape=None, alpha=1.0, depth_multiplier=1, dropout=1e-3, include_top=True, weights='imagenet', input_tensor=None, pooling=None, classes=1000) model .. parsed-literal:: Using TensorFlow backend. .. parsed-literal:: WARNING:tensorflow:From c:\python372_x64\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From c:\python372_x64\lib\site-packages\keras\backend\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`. .. parsed-literal:: .. code:: ipython3 model.name .. parsed-literal:: 'mobilenet_1.00_224' The model is stored here: .. code:: ipython3 import os os.listdir(os.path.join(os.environ.get('USERPROFILE', os.environ.get('HOME', '.')), ".keras", "models")) .. parsed-literal:: ['densenet121_weights_tf_dim_ordering_tf_kernels.h5', 'imagenet_class_index.json', 'mobilenet_1_0_224_tf.h5', 'mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_224.h5'] .. code:: ipython3 print(model.summary()) .. parsed-literal:: _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) (None, 224, 224, 3) 0 _________________________________________________________________ conv1_pad (ZeroPadding2D) (None, 225, 225, 3) 0 _________________________________________________________________ conv1 (Conv2D) (None, 112, 112, 32) 864 _________________________________________________________________ conv1_bn (BatchNormalization (None, 112, 112, 32) 128 _________________________________________________________________ conv1_relu (ReLU) (None, 112, 112, 32) 0 _________________________________________________________________ conv_dw_1 (DepthwiseConv2D) (None, 112, 112, 32) 288 _________________________________________________________________ conv_dw_1_bn (BatchNormaliza (None, 112, 112, 32) 128 _________________________________________________________________ conv_dw_1_relu (ReLU) (None, 112, 112, 32) 0 _________________________________________________________________ conv_pw_1 (Conv2D) (None, 112, 112, 64) 2048 _________________________________________________________________ conv_pw_1_bn (BatchNormaliza (None, 112, 112, 64) 256 _________________________________________________________________ conv_pw_1_relu (ReLU) (None, 112, 112, 64) 0 _________________________________________________________________ conv_pad_2 (ZeroPadding2D) (None, 113, 113, 64) 0 _________________________________________________________________ conv_dw_2 (DepthwiseConv2D) (None, 56, 56, 64) 576 _________________________________________________________________ conv_dw_2_bn (BatchNormaliza (None, 56, 56, 64) 256 _________________________________________________________________ conv_dw_2_relu (ReLU) (None, 56, 56, 64) 0 _________________________________________________________________ conv_pw_2 (Conv2D) (None, 56, 56, 128) 8192 _________________________________________________________________ conv_pw_2_bn (BatchNormaliza (None, 56, 56, 128) 512 _________________________________________________________________ conv_pw_2_relu (ReLU) (None, 56, 56, 128) 0 _________________________________________________________________ conv_dw_3 (DepthwiseConv2D) (None, 56, 56, 128) 1152 _________________________________________________________________ conv_dw_3_bn (BatchNormaliza (None, 56, 56, 128) 512 _________________________________________________________________ conv_dw_3_relu (ReLU) (None, 56, 56, 128) 0 _________________________________________________________________ conv_pw_3 (Conv2D) (None, 56, 56, 128) 16384 _________________________________________________________________ conv_pw_3_bn (BatchNormaliza (None, 56, 56, 128) 512 _________________________________________________________________ conv_pw_3_relu (ReLU) (None, 56, 56, 128) 0 _________________________________________________________________ conv_pad_4 (ZeroPadding2D) (None, 57, 57, 128) 0 _________________________________________________________________ conv_dw_4 (DepthwiseConv2D) (None, 28, 28, 128) 1152 _________________________________________________________________ conv_dw_4_bn (BatchNormaliza (None, 28, 28, 128) 512 _________________________________________________________________ conv_dw_4_relu (ReLU) (None, 28, 28, 128) 0 _________________________________________________________________ conv_pw_4 (Conv2D) (None, 28, 28, 256) 32768 _________________________________________________________________ conv_pw_4_bn (BatchNormaliza (None, 28, 28, 256) 1024 _________________________________________________________________ conv_pw_4_relu (ReLU) (None, 28, 28, 256) 0 _________________________________________________________________ conv_dw_5 (DepthwiseConv2D) (None, 28, 28, 256) 2304 _________________________________________________________________ conv_dw_5_bn (BatchNormaliza (None, 28, 28, 256) 1024 _________________________________________________________________ conv_dw_5_relu (ReLU) (None, 28, 28, 256) 0 _________________________________________________________________ conv_pw_5 (Conv2D) (None, 28, 28, 256) 65536 _________________________________________________________________ conv_pw_5_bn (BatchNormaliza (None, 28, 28, 256) 1024 _________________________________________________________________ conv_pw_5_relu (ReLU) (None, 28, 28, 256) 0 _________________________________________________________________ conv_pad_6 (ZeroPadding2D) (None, 29, 29, 256) 0 _________________________________________________________________ conv_dw_6 (DepthwiseConv2D) (None, 14, 14, 256) 2304 _________________________________________________________________ conv_dw_6_bn (BatchNormaliza (None, 14, 14, 256) 1024 _________________________________________________________________ conv_dw_6_relu (ReLU) (None, 14, 14, 256) 0 _________________________________________________________________ conv_pw_6 (Conv2D) (None, 14, 14, 512) 131072 _________________________________________________________________ conv_pw_6_bn (BatchNormaliza (None, 14, 14, 512) 2048 _________________________________________________________________ conv_pw_6_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_dw_7 (DepthwiseConv2D) (None, 14, 14, 512) 4608 _________________________________________________________________ conv_dw_7_bn (BatchNormaliza (None, 14, 14, 512) 2048 _________________________________________________________________ conv_dw_7_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_pw_7 (Conv2D) (None, 14, 14, 512) 262144 _________________________________________________________________ conv_pw_7_bn (BatchNormaliza (None, 14, 14, 512) 2048 _________________________________________________________________ conv_pw_7_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_dw_8 (DepthwiseConv2D) (None, 14, 14, 512) 4608 _________________________________________________________________ conv_dw_8_bn (BatchNormaliza (None, 14, 14, 512) 2048 _________________________________________________________________ conv_dw_8_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_pw_8 (Conv2D) (None, 14, 14, 512) 262144 _________________________________________________________________ conv_pw_8_bn (BatchNormaliza (None, 14, 14, 512) 2048 _________________________________________________________________ conv_pw_8_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_dw_9 (DepthwiseConv2D) (None, 14, 14, 512) 4608 _________________________________________________________________ conv_dw_9_bn (BatchNormaliza (None, 14, 14, 512) 2048 _________________________________________________________________ conv_dw_9_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_pw_9 (Conv2D) (None, 14, 14, 512) 262144 _________________________________________________________________ conv_pw_9_bn (BatchNormaliza (None, 14, 14, 512) 2048 _________________________________________________________________ conv_pw_9_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_dw_10 (DepthwiseConv2D) (None, 14, 14, 512) 4608 _________________________________________________________________ conv_dw_10_bn (BatchNormaliz (None, 14, 14, 512) 2048 _________________________________________________________________ conv_dw_10_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_pw_10 (Conv2D) (None, 14, 14, 512) 262144 _________________________________________________________________ conv_pw_10_bn (BatchNormaliz (None, 14, 14, 512) 2048 _________________________________________________________________ conv_pw_10_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_dw_11 (DepthwiseConv2D) (None, 14, 14, 512) 4608 _________________________________________________________________ conv_dw_11_bn (BatchNormaliz (None, 14, 14, 512) 2048 _________________________________________________________________ conv_dw_11_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_pw_11 (Conv2D) (None, 14, 14, 512) 262144 _________________________________________________________________ conv_pw_11_bn (BatchNormaliz (None, 14, 14, 512) 2048 _________________________________________________________________ conv_pw_11_relu (ReLU) (None, 14, 14, 512) 0 _________________________________________________________________ conv_pad_12 (ZeroPadding2D) (None, 15, 15, 512) 0 _________________________________________________________________ conv_dw_12 (DepthwiseConv2D) (None, 7, 7, 512) 4608 _________________________________________________________________ conv_dw_12_bn (BatchNormaliz (None, 7, 7, 512) 2048 _________________________________________________________________ conv_dw_12_relu (ReLU) (None, 7, 7, 512) 0 _________________________________________________________________ conv_pw_12 (Conv2D) (None, 7, 7, 1024) 524288 _________________________________________________________________ conv_pw_12_bn (BatchNormaliz (None, 7, 7, 1024) 4096 _________________________________________________________________ conv_pw_12_relu (ReLU) (None, 7, 7, 1024) 0 _________________________________________________________________ conv_dw_13 (DepthwiseConv2D) (None, 7, 7, 1024) 9216 _________________________________________________________________ conv_dw_13_bn (BatchNormaliz (None, 7, 7, 1024) 4096 _________________________________________________________________ conv_dw_13_relu (ReLU) (None, 7, 7, 1024) 0 _________________________________________________________________ conv_pw_13 (Conv2D) (None, 7, 7, 1024) 1048576 _________________________________________________________________ conv_pw_13_bn (BatchNormaliz (None, 7, 7, 1024) 4096 _________________________________________________________________ conv_pw_13_relu (ReLU) (None, 7, 7, 1024) 0 _________________________________________________________________ global_average_pooling2d_1 ( (None, 1024) 0 _________________________________________________________________ reshape_1 (Reshape) (None, 1, 1, 1024) 0 _________________________________________________________________ dropout (Dropout) (None, 1, 1, 1024) 0 _________________________________________________________________ conv_preds (Conv2D) (None, 1, 1, 1000) 1025000 _________________________________________________________________ act_softmax (Activation) (None, 1, 1, 1000) 0 _________________________________________________________________ reshape_2 (Reshape) (None, 1000) 0 ================================================================= Total params: 4,253,864 Trainable params: 4,231,976 Non-trainable params: 21,888 _________________________________________________________________ None .. code:: ipython3 len(model.layers) .. parsed-literal:: 93 Images ------ We collect images from `pixabay `__. Raw images ~~~~~~~~~~ .. code:: ipython3 from pyquickhelper.filehelper import unzip_files if not os.path.exists('simages'): os.mkdir('simages') files = unzip_files("data/dog-cat-pixabay.zip", where_to="simages") len(files), files[0] .. parsed-literal:: (31, 'simages\\cat-1151519__480.jpg') .. code:: ipython3 from mlinsights.plotting import plot_gallery_images plot_gallery_images(files[:2]); .. image:: search_images_keras_13_0.png .. code:: ipython3 from keras.preprocessing.image import array_to_img, img_to_array, load_img img = load_img('simages/cat-2603300__480.jpg') x = img_to_array(img) x.shape .. parsed-literal:: (480, 320, 3) .. code:: ipython3 import matplotlib.pyplot as plt plt.imshow(x / 255) plt.axis('off'); .. image:: search_images_keras_15_0.png `keras `__ implements optimized function to load and process images. Below the code with loads the images without modifying them. It creates an iterator which iterates as many times as we want. .. code:: ipython3 params = dict(rescale=1./255) I suggest trying without the parameter *rescale* to see the differences. The neural network expects numbers in ``[0, 1]`` not in ``[0, 255]``. .. code:: ipython3 from keras.preprocessing.image import ImageDataGenerator import numpy augmenting_datagen = ImageDataGenerator(**params) itim = augmenting_datagen.flow(x[numpy.newaxis, :, :, :]) # zip(range(0,2)) means to stop the loop after 2 iterations imgs = list(img[0] for i, img in zip(range(0,2), itim)) len(imgs), imgs[0].shape .. parsed-literal:: (2, (480, 320, 3)) .. code:: ipython3 plot_gallery_images(imgs); .. image:: search_images_keras_20_0.png But you can multiply the images. See `ImageDataGenerator `__ parameters to see what kind of modifications is implemented. .. code:: ipython3 augmenting_datagen_2 = ImageDataGenerator(rotation_range=40, channel_shift_range=9, **params) itim = augmenting_datagen_2.flow(x[numpy.newaxis, :, :, :]) imgs = list(img[0] for i, img in zip(range(0,10), itim)) plot_gallery_images(imgs); .. image:: search_images_keras_22_0.png Iterator on images ~~~~~~~~~~~~~~~~~~ We create an iterator, it considers every subfolder of images. We also need to rescale to size *(224, 224)* which is the size the loaded neural network ingests. .. code:: ipython3 flow = augmenting_datagen.flow_from_directory('.', batch_size=1, target_size=(224, 224), classes=['simages']) imgs = list(img[0][0] for i, img in zip(range(0,10), flow)) len(imgs), imgs[0].shape, type(flow) .. parsed-literal:: Found 62 images belonging to 1 classes. .. parsed-literal:: (10, (224, 224, 3), keras_preprocessing.image.directory_iterator.DirectoryIterator) .. code:: ipython3 plot_gallery_images(imgs); .. image:: search_images_keras_25_0.png How to get the image name? .. code:: ipython3 def get_current_index(flow): # The iterator is one step ahead. return (flow.batch_index + flow.n - 1) % flow.n def get_file_index(flow): n = get_current_index(flow) return flow.index_array[n] flow = augmenting_datagen.flow_from_directory('.', batch_size=1, target_size=(224, 224), classes=['simages'], shuffle=False) imgs = list((img[0][0], get_current_index(flow), flow.index_array[get_current_index(flow)], flow.filenames[get_file_index(flow)]) for i, img in zip(range(0,31), flow)) imgs[0][1:] .. parsed-literal:: Found 62 images belonging to 1 classes. .. parsed-literal:: (0, 0, 'simages\\cat-1151519__480.jpg') .. code:: ipython3 imgs[-1][1:] .. parsed-literal:: (30, 30, 'simages\\wolf-2865653__480.jpg') .. code:: ipython3 imgs = list((img[0][0], flow.filenames[get_current_index(flow)]) \ for i, img in zip(range(0,10), flow)) plot_gallery_images([_[0] for _ in imgs], [_[1] for _ in imgs]); .. image:: search_images_keras_29_0.png To keep the original order. .. code:: ipython3 flow = augmenting_datagen.flow_from_directory('.', batch_size=1, target_size=(224, 224), shuffle=False, classes=['simages']) imgs = list((img[0][0], flow.filenames[get_file_index(flow)]) \ for i, img in zip(range(0,7), flow)) plot_gallery_images([_[0] for _ in imgs], [_[1] for _ in imgs]); .. parsed-literal:: Found 62 images belonging to 1 classes. .. image:: search_images_keras_31_1.png .. code:: ipython3 len(flow) .. parsed-literal:: 62 Search among images ------------------- We use the class ``SearchEnginePredictionImages``. The idea of the search engine ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The deep network is able to classify images coming from a competition called `ImageNet `__ which was trained to classify different images. But still, the network has 88 layers which slightly transform the images into classification results. We assume the last layers contains information which allows the network to classify into objects: it is less related to the images than the content of it. In particular, we would like that an image with a daark background does not necessarily return images with a dark background. We reshape an image into *(224x224)* which is the size the network ingests. We propagate the inputs until the layer just before the last one. Its output will be considered as the *featurized image*. We do that for a specific set of images called the *neighbors*. When a new image comes up, we apply the same process and find the closest images among the set of neighbors. .. code:: ipython3 model = MobileNet(input_shape=None, alpha=1.0, depth_multiplier=1, dropout=1e-3, include_top=True, weights='imagenet', input_tensor=None, pooling=None, classes=1000) model .. parsed-literal:: .. code:: ipython3 from keras.models import Model output = model.layers[len(model.layers)-2].output model = Model(model.input, output) .. code:: ipython3 flow = augmenting_datagen.flow_from_directory('.', batch_size=1, target_size=(224, 224), classes=['simages'], shuffle=False) imgs = [img[0][0] for i, img in zip(range(0,31), flow)] .. parsed-literal:: Found 62 images belonging to 1 classes. .. code:: ipython3 outputs = [model.predict(im[numpy.newaxis, :, :, :]) for im in imgs] .. code:: ipython3 all_outputs = numpy.stack([o.ravel() for o in outputs]) We have the features. We build the neighbors. .. code:: ipython3 from sklearn.neighbors import NearestNeighbors knn = NearestNeighbors() knn.fit(all_outputs) .. parsed-literal:: NearestNeighbors(algorithm='auto', leaf_size=30, metric='minkowski', metric_params=None, n_jobs=None, n_neighbors=5, p=2, radius=1.0) We extract the neighbors for a new image. .. code:: ipython3 one_image = imgs[5] one_output = model.predict(one_image[numpy.newaxis, :, :, :]) .. code:: ipython3 score, index = knn.kneighbors([one_output.ravel()]) score, index .. parsed-literal:: (array([[0. , 0.22400763, 0.25415188, 0.2831644 , 0.29702211]]), array([[ 5, 28, 2, 1, 11]], dtype=int64)) We need to retrieve images for indexes stored in *index*. .. code:: ipython3 import os names = os.listdir("simages") names = [os.path.join("simages", n) for n in names] disp = [names[i] for i in index.ravel()] disp .. parsed-literal:: ['simages\\cat-2603300__480.jpg', 'simages\\schafer-dog-2669660__480.jpg', 'simages\\cat-1508613__480.jpg', 'simages\\cat-1192026__480.jpg', 'simages\\cat-2946028__480.jpg'] .. code:: ipython3 plot_gallery_images(disp); .. image:: search_images_keras_47_0.png Using one intermediate layer close to the output ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We do the same but with another code implemented in this module which does the same thing. .. code:: ipython3 model = MobileNet(input_shape=None, alpha=1.0, depth_multiplier=1, dropout=1e-3, include_top=True, weights='imagenet', input_tensor=None, pooling=None, classes=1000) model .. parsed-literal:: .. code:: ipython3 gen = ImageDataGenerator(rescale=1./255) iterimf = gen.flow_from_directory(".", batch_size=1, target_size=(224, 224), classes=['simages'], shuffle=False) .. parsed-literal:: Found 62 images belonging to 1 classes. .. code:: ipython3 from mlinsights.search_rank import SearchEnginePredictionImages se = SearchEnginePredictionImages(model, fct_params=dict(layer=len(model.layers) - 2), n_neighbors=5) .. code:: ipython3 se.fit(iterimf) se.features_.shape .. parsed-literal:: (62, 1000) .. code:: ipython3 se.features_.shape .. parsed-literal:: (62, 1000) .. code:: ipython3 list(se.metadata_)[:5] .. parsed-literal:: ['i', 'name'] .. code:: ipython3 se.metadata_.shape .. parsed-literal:: (62, 2) Let’s choose one image. .. code:: ipython3 name = se.metadata_.loc[5, "name"] name .. parsed-literal:: 'simages\\cat-2603300__480.jpg' .. code:: ipython3 img = load_img(name, target_size=(224, 224)) x = img_to_array(img) .. code:: ipython3 gen = ImageDataGenerator(rescale=1./255) iterim = gen.flow(x[numpy.newaxis, :, :, :], batch_size=1) .. code:: ipython3 score, ind, meta = se.kneighbors(iterim) .. code:: ipython3 score, ind, meta .. parsed-literal:: (array([0. , 0. , 0.22400763, 0.22400763, 0.25415188]), array([ 5, 36, 59, 28, 33], dtype=int64), i name 5 5 simages\cat-2603300__480.jpg 36 36 simages\category\cat-2603300__480.jpg 59 59 simages\category\shotlanskogo-2934720__480.jpg 28 28 simages\shotlanskogo-2934720__480.jpg 33 33 simages\category\cat-1508613__480.jpg) .. code:: ipython3 texts = ['original'] + [str(_) for _ in score] imgs = [name] + list(meta.name) plot_gallery_images(imgs, texts); .. image:: search_images_keras_62_0.png Using one intermediate layer close to the input ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: ipython3 se = SearchEnginePredictionImages(model, fct_params=dict(layer=1), n_neighbors=5) se.fit(iterimf) se.features_.shape .. parsed-literal:: (62, 151875) .. code:: ipython3 iterim = gen.flow(x[numpy.newaxis, :, :, :], batch_size=1) score, ind, meta = se.kneighbors(iterim) .. code:: ipython3 texts = ['original'] + [str(_) for _ in score] imgs = [name] + list(meta.name) plot_gallery_images(imgs, texts); .. image:: search_images_keras_66_0.png This is worse but expected. Going further ------------- The original neural network has not been changed and was chosen to be small (88 layers). Other options are available for better performances. The imported model can be also be trained on a classification problem if there is such information to leverage. Even if the model was trained on millions of images, a couple of thousands are enough to train the last layers. The model can also be trained as long as there exists a way to compute a gradient. We could imagine to label the result of this search engine and train the model on pairs of images ranked in the other. We can use the `pairwise transform `__ (example of code: `ranking.py `__). For every pair :math:`(X_i, X_j)`, we tell if the search engine should have :math:`X_i \prec X_j` (:math:`Y_{ij} = 1`) or the order order (:math:`Y_{ij} = 0`). :math:`X_i` is the features produced by the neural network : :math:`X_i = f(\Omega, img_i)`. We train a classifier on the database: .. math:: (f(\Omega, img_i) - f(\Omega, img_j), Y_{ij})_{ij} A training algorithm based on a gradient will have to propagate the gradient : :math:`\frac{\partial f}{\partial \Omega}(img_i) - \frac{\partial f}{\partial \Omega}(img_j)`.