Image Similarity Using UNET AutoEncoder And KNN

Hi, I recently worked on a conversational UI chatbot for a wedding card website, When we analyzed the wedding cards customer care support most of the user bring some sample of the wedding cards image and they ask the customer executive person to show similar or same wedding cards, but for customer executive person it is a tedious job to find some similar wedding card quickly due to that we decided why can we give this feature to the Chatbot So that chatbot can work on the sample card and give a similar image or same card within a fraction of second. So I started working on the image similarity.

At first, I gathered some image from the google image search and also some website using the scrapy tool and I started training the image with single autoencoder to get the latent representation of each image and using the latent representation we trained the KNN to cluster the latent represented image. So if you send a new image to encoder part it will give latent representation then it is sent to the already trained KNN and KNN will give the nearest neighbor image. This technique I got referenced from the below link

Finding Similar Images using Autoencoders

This link is awesome so take look out at it.

But, I didn’t get the best result using a single autoencoder.

So I started analyzing the wedding card type and pattern.

after the analyzation, I got some intuitive about the wedding card

At first, I classified the image data into 8groups.

Hindu wedding card

Muslim wedding card

Christian wedding card

Laser cutting wedding card

Floral wedding card

3d popup wedding card

Scroll wedding card

Box wedding card

I architected a classification model to classify this 8 label, then I architected 8 different autoencoders for this 8 label after I trained 8 KNN model to cluster the given latent represented data.

For example, If I sent a laser cutting wedding card image to the classification model it will classify as laser cutting cards then, it will choose pre-trained encoder part and send the image to the laser cutting encoder model and get the latent represented data after it is sent to the KNN it will give the similar image of laser cutting card.

So I downloaded the image for each label around 100 images, then I trained the 100 images to the autoencoder, for example, I trained 100 box wedding card image to one autoencoder. So it has a latent representation of all the 100 box wedding card, If I send a box wedding card which is in red color and small box then KNN will give first 10 similar images which also red color and small box design.

Here I used Unet as an autoencoder , but In above link they used a normal convolution autoencoder.

vigneshgig/similiarimage

In above GitHub link, you can find dataset creating notebook and UNET autoencoder notebook file but I haven't included the xception classification code.

Note: This Xception code is not for this problem, but it is easy to modify or you can get many tutorials regarding xception classification problem.

from keras.applications.xception import Xception, preprocess_input
from keras.preprocessing import image
from keras.models import Model
from keras.layers import Input
from keras.layers import Dense, GlobalAveragePooling2D
from keras import backend as K
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import Flatten
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers import BatchNormalization
import keras
import numpy as np
import glob
import cv2
import os
import matplotlib.pyplot as plt
import h5py
hdf5_path = '/content/drive/My Drive/monogramdataset.hdf5'
hdf5_pathtest = '/content/drive/My Drive/monogramdatasetvalidation.hdf5'
hdf5_file = h5py.File(hdf5_path, "r")
hdf5_filetest = h5py.File(hdf5_pathtest, "r")
# reshape to be [samples][pixels][width][height]
X_train = hdf5_file.get('train_img').value
y_train = hdf5_file.get('train_labels').value
X_val  = hdf5_filetest.get('val_img').value
y_val  = hdf5_filetest.get('val_labels').value
X_test  = hdf5_filetest.get('test_img').value
y_test  = hdf5_filetest.get('test_labels').value
# X_train = X_train.reshape(X_train.shape[2],X_train.shape[1],X_train.shape[0])
# X_test = X_test.reshape(X_test.shape[2],X_test.shape[1],X_test.shape[0])
hdf5_file.close()
X_train = X_train.reshape(X_train.shape[0], 299, 299,3).astype('float32')
X_test = X_test.reshape(X_test.shape[0], 299, 299,3).astype('float32')
X_val  = X_val.reshape(X_val.shape[0], 299, 299,3).astype('float32')
# X_train = X_train.reshape(X_train.shape[0], 299, 299,1).astype('float32')
# X_test = X_test.reshape(X_test.shape[0], 299, 299,1).astype('float32')
# X_val  = X_val.reshape(X_val.shape[0], 299, 299,1).astype('float32')
# create base pre-trained model
from keras.applications.xception import Xception
# from keras.applications.inception_resnet_v2 import InceptionResNetV2
base_model = Xception(weights='imagenet', include_top=False)
#add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(200,activation='elu')(x)
x = Dropout(0.4)(x)
x = Dense(170,activation='elu')(x)
predictions = Dense(157,activation='hard_sigmoid')(x)
model = Model(inputs=base_model.input, outputs=predictions)
# # we use SGD with a low learning rate
from keras.optimizers import Nadam
from keras.optimizers import adam
model.compile(optimizer=Nadam(lr=0.0001), loss='binary_crossentropy',metrics=['accuracy'])
# model.compile(optimizer='adam', loss='binary_crossentropy',metrics=['accuracy'])
# # we train our model again (this time fine-tuning the top 2 inception blocks
# # alongside the top Dense layers
# model.fit_generator(...)
model.fit(X_train, y_train, validation_data=(X_val, y_val),shuffle="batch" ,epochs=10, batch_size=10,verbose=1)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0,batch_size=10)
print("Baseline Error: %.2f%%" % (100-scores[1]*100))
# # serialize model to JSON
try:
    model_json = model.to_json()
    with open("/content/drive/My Drive/monogrammodelxceptioneludrop299.json", "w") as json_file:
        json_file.write(model_json)
except:
    pass
# serialize weights to HDF5
model.save_weights("/content/drive/My Drive/monogrammodelxceptioneludrop299.h5")
print("Saved model to disk")

and also I have given a link for the similarimage folder which contains the pre-trained model and dataset make use of it.

similiar_image_model

Thanks for reading my blogs………..