Image Search Guide

The Image Search project is an AI-based search solution that captures similarity among all of a customer's previously indexed images by taking a reference image. It completes its flow primarily through 3 applications:

• Image-search extension
• Commerce
• Frontend (Next or Python)

Setup

Image-Search Extension

The flow responsible for indexing data to Elasticsearch on the Commerce side also transfers data to the image search extension. For customers being set up for the first time, after the necessary checks are performed, a full index must be initiated. This way, the product-image relationship will be recorded in the extension, and through background jobs, embeddings (embedded data vectors) belonging to the images will be generated and stored in the pgvector database.

For setup, every stable-tagged version of the image search extension must be installed with the parameters below. The parameter set is listed below.

SEC_API_AUTH_PASSWORD	20-character combination of uppercase, lowercase letters and numbers (should be defined as a secret)
API_AUTH_USERNAME	15-character combination of uppercase, lowercase letters and numbers.

Commerce

It essentially fulfills two tasks:

1. Ensuring that products are indexed to the extensions.
2. Mediating the forwarding of requests coming from the Frontend application to the extensions.

The settings that need to be entered are below:

Environment Variables:

SEARCH_DYNAMIC_FILTER_ACTIVE	True	Setting that must be set to true so that the product IDs returned in the response as a result of image-search can be used in filtering operations.

Dynamic Settings:

MAX_IMAGE_DIMENSIONS	{ "width": 3000, "height": 3000 }	Setting specifying the maximum dimensions of the image that can be uploaded.
EXTERNAL_SEARCH_EXTENSION_CONFIG	{ "host": "{extension_host}", "klass": "omnishop.search.libs.external_search.extensions.similarity_search.client.SimilaritySearchExtensionClient", "limit": 20, "enabled": false, "password": "{extension_password}", "username": "{extension_username}" }	Settings that need to be defined for extension configuration.

EXTERNAL_SEARCH_EXTENSION_CONFIG:

host	The URL used to connect to the extension.
klass	The class to be used for similarity search. (SimilaritySearchExtensionClient)
limit	Parameter specifying the maximum number of products that will be returned in a request made to list similar products. The limit value is added to the request sent to the extension.
enabled	Flag indicating whether the image search feature is active or not (true/false).
password	The 20-character-long password required for access mentioned in the extension section.
username	The 15-character-long username required for access mentioned in the extension section.

If the Commerce project is "old commerce," ImageSearchView must be added under shomnipro/urls.py. You can find a sample pull request here.

FE-Next

On the Next side, the system works via a plugin structure. You can refer to the link below. https://bitbucket.org/akinonteam/pz-similar-products/src/main/

FE-Python

This may vary depending on the brand's filtering and product item designs. The general structure is available in the files below and can be used as a reference:

https://bitbucket.org/akinonteam/fashfed/src/master/templates/partials/modal/search/

https://bitbucket.org/akinonteam/fashfed/src/master/templates/product/product-sliders/index.html (pdp button)

https://bitbucket.org/akinonteam/fashfed/src/master/templates/partials/search/index.html (search button)

https://bitbucket.org/akinonteam/fashfed/src/master/templates/partials/index.js (localStorage check)

Another example from the Koton project: https://bitbucket.org/%7B489c1f97-f93a-4877-a1d2-9971a8bc6132%7D/%7Be75b7ee1-b893-471a-ab51-97308cadec92%7D/pull-requests/1225/diff

1. API Request: getSimilarProducts

This function retrieves similar products using a cropped image (POST) or an image URL (GET).

async getSimilarProducts(image, isCropped, isProduct, isReco) {
  this.loading = true;
  const apiUrl = '/image-search/';
  
  const queryParams = `?limit=20${
    isCropped ? '' : `&url=${encodeURIComponent(image)}`
  }${
    (isProduct && !isCropped) ? `&excluded_product_ids=${this.productPk}` : ''
  }${
    (!isCropped && this.searchText) ? `&text=${this.searchText}` : ''
  }`;

  const response = await fetch(`${apiUrl}${queryParams}`, {
    method: isCropped ? 'POST' : 'GET',
    headers: {
      'Content-Type': 'application/json',
      'x-csrftoken': $(window.GLOBALS.csrf_token).val()
    },
    ...(isCropped && { 
      body: JSON.stringify({ 
        image, 
        ...(isProduct && { excluded_product_ids: [+this.productPk] }),
        ...(this.searchText && { text: this.searchText })
      }) 
    })
  });

  const data = await response.json();

  if (data.image?.length) {
    this.showError(data.image[0]);
  } else {
    this.processSimilarProductsResponse(data, isReco);
  }
}

2. Processing the API Response: processSimilarProductsResponse

This function parses the API response, extracts the product IDs, and triggers the process of fetching the filtered list.

processSimilarProductsResponse(response, isReco) {
  const productIds = this.extractProductIds(response);

  if (!productIds.length) {
    return this.showNoResults();
  }

  const filterObject = { 
    'products.pk': productIds, 
    default_sorting_deactivated: true 
  };

  this.base64Encoded = btoa(JSON.stringify(filterObject));
  this.fetchFilteredList(isReco);
}

3. Fetching the Filtered Product List: fetchFilteredList

Fetches the final product list with the applied filters and optionally updates the recommendation section.

async fetchFilteredList(isReco) {
  this.updateSelectedFilters();
  const listUrl = `/list/${this.selectedFilters.toString() ? `?${this.selectedFilters.toString()}` : ''}`;

  const response = await fetch(listUrl, {
    headers: {
      'x-search-dynamic-filter': this.base64Encoded,
      'Accept': 'application/json',
      'Content-Type': 'application/json'
    }
  });

  const data = await response.json();

  if (isReco && data.products.length) {
    this.similarRecoList.innerHTML = data.products
      .slice(0, 3)
      .map(p => `<a href="${p.absolute_url}"><img src="${p.productimage_set?.[0]?.image}"/></a>`)
      .join('');
  } else {
    this.renderProductList(data);
  }
}

Mobile App

Requirements

This feature is available for projects using env5.3 or higher versions.

Platform Setup

Android Configuration

Add the required permissions to the Akinon.json file for Android:

{
  "android": {
    "permissions": [
      "android.permission.CAMERA"
    ]
  }
}

iOS Configuration

Add the required permissions to the Akinon.json file for iOS:

{
  "ios": {
    "infoPlist": {
      "NSCameraUsageDescription": "This app requires camera access to take photos for image search.",
      "NSPhotoLibraryUsageDescription": "This app requires photo library access to select photos for image search."
    }
  }
}

Project Setup

1. Default Styles and Components

View the diff of the default styles, texts, and components that have been added here: diff

2. Icon Configuration

Add the following new icons with the default names below to the Icomoon icon file:

• similar-items
• camera
• gallery
• crop

3. API Endpoint Configuration

Add the image search endpoint to the src/integrationMaps/urls.json file. Make sure this endpoint is defined in Commerce or Zero, and that the list endpoint works with the x-search-dynamic-filter header:

{
  "IMAGE_SEARCH": "/image-search/"
}

Implementation Guide

Search Page Implementation

Import and use the ImageSearch component in the src/pages/search/index.js file:

import ImageSearch from "_components/imageSearch"

const searchGrid = (pageContext) => {
  return (
    <ImageSearch
      tooltip
      tooltipPlacement="bottom"
      iconName="similar-items"
      iconClassName="imageSearch.gridElements.imageSearchIcon"
      iconContainerClassName="imageSearch.gridElements.imageSearchIconContainer"
    />
  )
}

Product List Page Implementation

Import and use the ImageSearchResult component in the src/pages/productList/index.js file:

import ImageSearchResult from "_components/imageSearch/result";

const productListGrid = (pageContext) => {
  return (
    <ProductList
      ListHeaderComponent={
        <ImageSearchResult 
          navigationType="replace" 
          imageContainerSize={150} 
        />
      }
    />
  )
}

Product Detail Page Implementation

Implement the image search functionality in the src/pages/productDetail/index.js file:

import ImageSearch from "_components/imageSearch"

const ProductDetailGrid = (pageContext) => {
  const { product } = pageContext;
  
  return (
    <ImageSearch
      iconContainerClassName="imageSearch.gridElements.imageSearchIconContainerPd"
      iconClassName="imageSearch.gridElements.imageSearchIconPd"
      tooltip={false}
      onPress={(params) => {
        params.imageSearchByUrl({
          imageUrl: product.images[0].image,
          excludedProductIds: product.pk
        })
      }}
    />
  )
}

Enabling for Use

Activating Commerce

After the setup is completed, enabled=true must be updated on Commerce, and it should be verified that this update is reflected in the extension following a product update. To do this, the following request can be sent to the extension:

curl --location 'https://EXTENSION_URL/api/v1/products/distribution' \
--header 'Authorization: Basic base64(UN:PW)'

Here, at least 1 record with a "SUCCESSFUL" status should be seen in a view similar to the following:

 "groups": [
       {
           "status": "SUCCESSFUL",
           "retryCount": 0,
           "count": 9
       },
       {
           "status": "WAITING",
           "retryCount": 0,
           "count": 5
       },
       {
           "status": "WORKING",
           "retryCount": 0,
           "count": 2
       }   ],
   "imageCount": 16,
   "productCount": 4,
   "productImageCount": 14,
   "productImageCorrelation": 3
}

Starting the Full Index

After the steps above have been completed successfully, a full index must be initiated by Commerce so that the flow of all products/images to the extension is ensured. This process, which is performed only during the initial setup, may take anywhere from a few hours to a few days depending on the size of the data. The final status of the full indexing can be tracked both from the metrics endpoint on the project's Omnitron:

curl --location 'https://OMNITRON_URL/api/v1/remote/1/metrics/' \
--header 'Authorization: Token OMNITRON_TOKEN

and also from the extension endpoint above. As the extension processes the images, similarity searches coming from the FE will start to yield results.

Checks in Observation Mode

Before the feature is fully released to the end customer, checks must be performed in observation mode. In a deployment made in this mode, the end customer cannot directly see the new features, but once "enable_image_search=true" is added to local storage in the browser, the features become available for use. Comprehensive checks should be performed both for queries made directly with an existing product image and for queries made via cropping, covering query speed, similar product checks, and whether errors occur. The product must NEVER be released directly to the end customer without these checks being performed.

Release to the End Customer

After the checks are completed, the FE applications need to make sure the feature is released to the end customer either by issuing a deploy or by making a parameter change.