> For the complete documentation index, see [llms.txt](https://docs.akinon.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.akinon.com/technical-guides/image-search-guide.md). # Image Search Guide The Image Search project is an AI-based search solution that captures similarity among all of a customer's previously indexed images by taking a reference image. It completes its flow primarily through 3 applications: * **Image-search extension** * **Commerce** * **Frontend (Next or Python)** ## **Setup** ### **Image-Search Extension** The flow responsible for indexing data to Elasticsearch on the Commerce side also transfers data to the image search extension. For customers being set up for the first time, after the necessary checks are performed, a full index must be initiated. This way, the product-image relationship will be recorded in the extension, and through background jobs, embeddings (embedded data vectors) belonging to the images will be generated and stored in the **pgvector** database. For setup, every stable-tagged version of the image search extension must be installed with the parameters below. The parameter set is listed below. | `SEC_API_AUTH_PASSWORD` | *20-character combination of uppercase, lowercase letters and numbers (should be defined as a secret)* | | ----------------------- | ------------------------------------------------------------------------------------------------------ | | `API_AUTH_USERNAME` | *15-character combination of uppercase, lowercase letters and numbers.* | ### **Commerce** It essentially fulfills two tasks: 1. Ensuring that products are indexed to the extensions. 2. Mediating the forwarding of requests coming from the Frontend application to the extensions. The settings that need to be entered are below: **Environment Variables:** | `SEARCH_DYNAMIC_FILTER_ACTIVE` | True | Setting that must be set to true so that the product IDs returned in the response as a result of image-search can be used in filtering operations. | | ------------------------------ | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------- | **Dynamic Settings:** | `MAX_IMAGE_DIMENSIONS` | `{ "width": 3000, "height": 3000 }` | Setting specifying the maximum dimensions of the image that can be uploaded. | | ---------------------------------- | ----------------------------------- | ---------------------------------------------------------------------------- | | `EXTERNAL_SEARCH_EXTENSION_CONFIG` | See the JSON example below. | Settings that need to be defined for extension configuration. | **EXTERNAL\_SEARCH\_EXTENSION\_CONFIG:** This setting holds an `extensions` list. Each item configures one external search extension; you may register more than one (for example, a similarity-search extension that serves queries and an embedding-store extension that only handles indexing). Image search is considered enabled when **at least one** extension in the list has `"enabled": true`; otherwise the `/image-search/` endpoint returns `404`. ```json { "extensions": [ { "enabled": true, "klass": "omnishop.search.libs.external_search.extensions.similarity_search.client.SimilaritySearchExtensionClient", "host": "{extension_host}", "username": "{extension_username}", "password": "{extension_password}", "limit": 20, "kwargs": { "multimodal_search_enabled": false } } ] } ``` | `enabled` | Flag indicating whether this extension is active (`true`/`false`). Image search runs when any extension in the list is enabled. | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `klass` | The extension class to use. `SimilaritySearchExtensionClient` serves the image-search queries; `EmbeddingStoreExtensionClient` handles the embedding/indexing side only. | | `host` | The URL used to connect to the extension. | | `username` | The 15-character-long username required for access mentioned in the extension section. | | `password` | The 20-character-long password required for access mentioned in the extension section. | | `limit` | Maximum number of similar products returned per request (default `20`). The value is forwarded to the extension as the `limit` parameter and caps how many results the search yields. | | `kwargs` | Optional extra flags. `multimodal_search_enabled` (default `false`) enables text-refined image search: only when it is `true` is the optional `text` keyword forwarded to the extension. | {% hint style="info" %} The legacy flat format (`host`, `klass`, `limit`, … at the top level, without the `extensions` list) is no longer used. Configurations must be provided under the `extensions` list shown above. {% endhint %} {% hint style="info" %} If the Commerce project is "old commerce," **ImageSearchView** must be added under **shomnipro/**[**urls.py**](http://urls.py/). You can find a sample pull request [here](https://bitbucket.org/akinonteam/occasion/pull-requests/1011). {% endhint %} ### **FE-Next** On the Next side, the system works via a plugin structure. You can refer to the link below.\ ### **FE-Python** This may vary depending on the brand's filtering and product item designs. The general structure is available in the files below and can be used as a reference: (pdp button) (search button) (localStorage check) Another example from the Koton project:\ #### **1. API Request: `getSimilarProducts`** This function retrieves similar products using a cropped image (POST) or an image URL (GET). ```javascript async getSimilarProducts(image, isCropped, isProduct, isReco) { this.loading = true; const apiUrl = '/image-search/'; const queryParams = `?limit=20${ isCropped ? '' : `&url=${encodeURIComponent(image)}` }${ (isProduct && !isCropped) ? `&excluded_product_ids=${this.productPk}` : '' }${ (!isCropped && this.searchText) ? `&text=${this.searchText}` : '' }`; const response = await fetch(`${apiUrl}${queryParams}`, { method: isCropped ? 'POST' : 'GET', headers: { 'Content-Type': 'application/json', 'x-csrftoken': $(window.GLOBALS.csrf_token).val() }, ...(isCropped && { body: JSON.stringify({ image, ...(isProduct && { excluded_product_ids: [+this.productPk] }), ...(this.searchText && { text: this.searchText }) }) }) }); const data = await response.json(); if (data.image?.length) { this.showError(data.image[0]); } else { this.processSimilarProductsResponse(data, isReco); } } ``` #### **2. Processing the API Response: `processSimilarProductsResponse`** This function parses the API response, extracts the product IDs, and triggers the process of fetching the filtered list. ```javascript processSimilarProductsResponse(response, isReco) { const productIds = this.extractProductIds(response); if (!productIds.length) { return this.showNoResults(); } const filterObject = { 'products.pk': productIds, default_sorting_deactivated: true }; this.base64Encoded = btoa(JSON.stringify(filterObject)); this.fetchFilteredList(isReco); } ``` #### **3. Fetching the Filtered Product List: `fetchFilteredList`** Fetches the final product list with the applied filters and optionally updates the recommendation section. ```javascript async fetchFilteredList(isReco) { this.updateSelectedFilters(); const listUrl = `/list/${this.selectedFilters.toString() ? `?${this.selectedFilters.toString()}` : ''}`; const response = await fetch(listUrl, { headers: { 'x-search-dynamic-filter': this.base64Encoded, 'Accept': 'application/json', 'Content-Type': 'application/json' } }); const data = await response.json(); if (isReco && data.products.length) { this.similarRecoList.innerHTML = data.products .slice(0, 3) .map(p => ``) .join(''); } else { this.renderProductList(data); } } ``` ### **Mobile App** #### **Requirements** This feature is available for projects using `env5.3` or higher versions. #### **Platform Setup** **Android Configuration** Add the required permissions to the Akinon.json file for Android: ```json { "android": { "permissions": [ "android.permission.CAMERA" ] } } ``` **iOS Configuration** Add the required permissions to the Akinon.json file for iOS: ```json { "ios": { "infoPlist": { "NSCameraUsageDescription": "This app requires camera access to take photos for image search.", "NSPhotoLibraryUsageDescription": "This app requires photo library access to select photos for image search." } } } ``` #### **Project Setup** **1. Default Styles and Components** View the diff of the default styles, texts, and components that have been added here: [diff](https://abp.akinon.net/upgrade-helper?from=v5.4.2\&to=v5.4.3) **2. Icon Configuration** Add the following new icons with the default names below to the Icomoon icon file: * `similar-items` * `camera` * `gallery` * `crop` **3. API Endpoint Configuration** Add the image search endpoint to the **src/integrationMaps/urls.json** file. Make sure this endpoint is defined in Commerce or Zero, and that the list endpoint works with the **x-search-dynamic-filter** header: ```json { "IMAGE_SEARCH": "/image-search/" } ``` #### **Implementation Guide** **Search Page Implementation** Import and use the **ImageSearch** component in the **src/pages/search/index.js** file: ```javascript import ImageSearch from "_components/imageSearch" const searchGrid = (pageContext) => { return ( ) } ``` **Product List Page Implementation** Import and use the **ImageSearchResult** component in the **src/pages/productList/index.js** file: ```javascript import ImageSearchResult from "_components/imageSearch/result"; const productListGrid = (pageContext) => { return ( } /> ) } ``` **Product Detail Page Implementation** Implement the image search functionality in the **src/pages/productDetail/index.js** file: ```javascript import ImageSearch from "_components/imageSearch" const ProductDetailGrid = (pageContext) => { const { product } = pageContext; return ( { params.imageSearchByUrl({ imageUrl: product.images[0].image, excludedProductIds: product.pk }) }} /> ) } ``` ## **Enabling for Use** ### **Activating Commerce** After the setup is completed, `enabled=true` must be updated on Commerce, and it should be verified that this update is reflected in the extension following a product update. To do this, the following request can be sent to the extension: ``` curl --location 'https://EXTENSION_URL/api/v1/products/distribution' \ --header 'Authorization: Basic base64(UN:PW)' ``` Here, at least 1 record with a "SUCCESSFUL" status should be seen in a view similar to the following: ```json "groups": [ { "status": "SUCCESSFUL", "retryCount": 0, "count": 9 }, { "status": "WAITING", "retryCount": 0, "count": 5 }, { "status": "WORKING", "retryCount": 0, "count": 2 } ], "imageCount": 16, "productCount": 4, "productImageCount": 14, "productImageCorrelation": 3 } ``` ### **Starting the Full Index** After the steps above have been completed successfully, a *full index* must be initiated by Commerce so that the flow of all products/images to the extension is ensured. This process, which is performed only during the initial setup, may take anywhere from a few hours to a few days depending on the size of the data. The final status of the full indexing can be tracked both from the metrics endpoint on the project's Omnitron: ``` curl --location 'https://OMNITRON_URL/api/v1/remote/1/metrics/' \ --header 'Authorization: Token OMNITRON_TOKEN ``` and also from the extension endpoint above. As the extension processes the images, similarity searches coming from the FE will start to yield results. ### **Checks in Observation Mode** Before the feature is fully released to the end customer, checks must be performed in observation mode. In a deployment made in this mode, the end customer cannot directly see the new features, but once "enable\_image\_search=true" is added to local storage in the browser, the features become available for use. Comprehensive checks should be performed both for queries made directly with an existing product image and for queries made via cropping, covering query speed, similar product checks, and whether errors occur. The product must NEVER be released directly to the end customer without these checks being performed. ### **Release to the End Customer** After the checks are completed, the FE applications need to make sure the feature is released to the end customer either by issuing a deploy or by making a parameter change. ## **Search Scope, Sorting & Troubleshooting** This section explains which products image search scans, whether that scope can be narrowed, how results are ordered, and how to investigate when the results differ from what is expected. ### **Which products image search scans** Image search does **not** scan the entire raw catalog. It scans the products that are indexed to the storefront's Elasticsearch catalog — the same set that powers normal text search and the `/list/` endpoint. Whenever a product is (re)indexed to Elasticsearch, the same change is forwarded asynchronously to the image-search extension; when a product is removed from the Elasticsearch index, it is removed from image search as well. A product is part of the image-search pool only when **all** of the following are true: * It is **listable**. * It is **active** and **not hidden**. * It has a **price**. * It is **in stock** — unless `PRODUCT_STOCK_OUT_VISIBILITY_ENABLED` is enabled, in which case out-of-stock products are also indexed and therefore searchable. If any of these conditions fails, the product is removed from the image-search index (the same way it drops out of normal listing/search). In addition, the product must have at least one **active product image**. Only active product images are sent to the extension to generate embeddings, so a product with no active image has nothing to match against and will never appear in results — even if it otherwise satisfies the conditions above. Indexing runs asynchronously (a background job triggered after each catalog index), so immediately after a catalog change there can be a short delay before image search reflects it. **Summary for the partner's question:** the scanned set is the Elasticsearch-indexed catalog (listable, active, priced, non-hidden, and in stock unless stock-out visibility is enabled) intersected with products that have an active image. Out-of-stock products are included only when `PRODUCT_STOCK_OUT_VISIBILITY_ENABLED` is on. ### **Can the scope be limited by channel or category?** The image-search request itself **cannot** be scoped by channel or category. The `/image-search/` endpoint accepts only the image (an image URL for `GET`, or a base64 image for `POST`), an optional `text` keyword (when multimodal search is enabled), and `excluded_product_ids`. No channel or category filter is sent to the similarity extension, and there is no per-channel similarity index. Scoping instead happens on the **second step** of the flow. The extension returns a list of product IDs, and the storefront feeds those IDs into the normal `/list/` endpoint through the `x-search-dynamic-filter` header (as `products.pk`). All the usual listing rules apply at that point: * Channel catalog membership. * `SEARCH_PRE_FILTER_KWARGS` (for example, restricting results to a category subtree). * Any category or attribute filters already active on the page. This `/list/` pass is what effectively narrows what the shopper finally sees. A product returned by the similarity engine can still be filtered out here if it is not part of the active channel or category listing. For this step to work, `SEARCH_DYNAMIC_FILTER_ACTIVE=True` must be set so the returned product IDs can be used as a dynamic filter. The only narrowing available directly on the image-search request is `excluded_product_ids`, which drops specific products from a single request (for example, excluding the product the shopper is currently viewing). ### **How results are ordered** Ordering is decided **entirely by the external similarity engine**, based on visual similarity to the uploaded image (and, when multimodal search is enabled, the optional `text` keyword). The storefront returns the engine's product IDs in the exact order received; it does **not** compute, expose, or re-rank by any similarity score, and no score is available to the storefront. To preserve that similarity order through the final `/list/` call, the request sets `default_sorting_deactivated: true`. This disables the storefront's default sort options so the products are shown in the engine's ranking rather than being re-sorted (for example, by price or newness). The number of results is capped by the extension's `limit` setting (default `20`). ### **Troubleshooting: why a product does or does not appear** When results differ from what is expected, work through this checklist: 1. **Is image search enabled?** If no extension in `EXTERNAL_SEARCH_EXTENSION_CONFIG` has `"enabled": true`, the `/image-search/` endpoint returns `404` and nothing is indexed. Confirm the setting and that a full index has completed (see *Starting the Full Index*). 2. **Does the product meet the indexing conditions?** A product that is not listable, not active, hidden, has no price, or is out of stock (while `PRODUCT_STOCK_OUT_VISIBILITY_ENABLED` is off) is removed from the image-search index and will not appear. 3. **Does the product have an active image?** Only active product images are embedded. A product with no active image cannot be matched. 4. **Has indexing caught up?** Indexing is asynchronous. After a recent catalog change or first-time setup, allow time for the background job (and, for the embedding store, the embedding generation) to finish. A failed or never-run reindex leaves the extension's data stale. 5. **Is the product being filtered out by the `/list/` step?** Even when the engine returns a product, the final `/list/` pass applies channel catalog, `SEARCH_PRE_FILTER_KWARGS`, and category filters. A product outside the active channel or category will not be shown. Confirm `SEARCH_DYNAMIC_FILTER_ACTIVE=True`. 6. **Was the product explicitly excluded?** Check whether its ID is in `excluded_product_ids` for the request. 7. **Unexpected or off-topic results (for example, shoes when a shirt was uploaded)?** Result relevance is determined by the external similarity engine, not by the storefront. A tightly cropped, single-item image yields more focused results; a busy or multi-item image can match a broader mix. If `text` is sent to refine results but `multimodal_search_enabled` is `false`, the keyword is silently ignored. 8. **Is the result count lower than expected?** Only the top `limit` products (default `20`) are returned by the extension, and the `/list/` filters may then remove some of those. 9. **Cross-channel or cross-category products appearing?** Because the similarity engine is not channel/category scoped, only the `/list/` filters restrict results. If those filters are not configured for the channel, products from other channels or categories in the shared index can appear. 10. **Image rejected before search?** Uploads must be a valid image (JPEG, PNG, JPG, WEBP), within `MAX_IMAGE_DIMENSIONS` (default `2000x2000` px), and within the maximum upload size. Otherwise the request returns a validation error instead of results. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.akinon.com/technical-guides/image-search-guide.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.