Search Structure
This document provides a comprehensive guide on the features and functionalities, and standard operating instructions for the Commerce Search Structure.
Search Structure
The search functionality on the Commerce side is offered through the "/list" endpoint. Triggering the search structure involves sending a request to this endpoint. The Commerce search structure relies on the usage of "ElasticSearch". When the application is initialized, the relevant search/elastic settings are provided to Django via apps.py. The corresponding view and method can be found at omnishop.search.resources.views.FacetView.get.
FacetView Get:
Acceptable Parameters for the Request
Possible Parameters in the Request for the respective endpoint:
search_text: The keyword to search for.
extra_search_param: Used for filtering based on fields such as in_stock, price, basket_offer_ids, category_ids, collection_ids.
index_name: The index to search in Elasticsearch.
default_page_size: The number of products for pagination.
elasticsearch_value: Used to select a sorter based on the elasticsearch_value found in Sort Options in the database.
extraction_strategy: The method used for extracting products from the response received from Elasticsearch.
inventory_strategy: Specifies which price inventory to use in shops with multiple inventories."
After the request is made, the search result is first checked to see if the same search has been previously performed using both the local Redis and default Redis servers. If a previous search with the same parameters has been made, the result is quickly retrieved from the cache. To achieve this, after the request is received and the "get" method is called, the "_get" method is invoked to check if the specified search exists in the cache. An example is provided below:
The expiration durations for the cache layers are as follows:
L1 Cache Layer (Local Redis): 60 seconds
L2 Cache Layer (Shared Redis): 120 seconds
Search Get Cache:
When a search is received that has not been previously performed, the cache layer will not have the result available, and the normal search process will be initiated. In this process, the Elasticsearch service is utilized. The SearchService in Omnishop > Search > services.py is responsible for handling this. Elasticsearch does not store the actual products and categories themselves. The necessary fields for querying include: in_stock, price, basket_offer_ids, category_ids, collection_ids.
Possible Search Params:
Sending a Request to ElasticSearch
When there is no data stored in the cache and a new search request is received, as mentioned above, no result can be obtained from the cache layer. Therefore, the normal search flow is initiated. In this process, the ElasticSearch service, as mentioned earlier, is involved. A request is sent to ElasticSearch to retrieve the relevant data. The parameters included in this request, along with their descriptions, are as follows:
index_name: The value of the query_param or the ES_INDEX value in the settings is used. If specified in the settings, the static ES_INDEX value is used, such as ES_INDEX="products".
inventory_strategy: The value of the query_param or the inventory_strategy of the basket is used. If the inventory_strategy is provided, the price field is replaced with price_1234, where 1234 is the price_list_id.
extraction_strategy: The value is taken from the query_param in the request. If the extraction_strategy value in the request is "None", the "get_search_product_extraction_strategy" method in omnishop.search.strategies is called. If defined in the settings, it takes the value of "SEARCH_PRODUCT_EXTRACTION_STRATEGY", otherwise, it takes the value of "omnishop.search.strategies.DefaultProductExtractionStrategy".
Inventory and Extraction Strategy:
Default Extraction Strategy:
In the next step, the Facet Fields are retrieved from Elasticsearch. These Facets essentially represent the information that can be used for filtering on the list screens of the shop websites. For example, categories, size, material, stock status, price, etc.
sorter: The value of the query_param or the default SortOption object found in the database is used. The elasticsearch_value field of the SortOption is the parameter used when making the request to Elasticsearch.
Filtering Flow on the Shop Screen
The Facet Fields and Sort Options are obtained during the execution flow of the "._search" method in the Search > Services > SearchService.
During the flow, the "._search" method is executed by the "Search Service" to retrieve the options for filtering on the list screen of shop websites. The Facet Fields are fetched from ElasticSearch for obtaining these filters. The "get_facet_fields" method is responsible for retrieving the facets.
When executing the method, the "facet_constructor" checks if the facet_field has any attributes, and if so, it configures the facets accordingly (e.g., aggregations, field, extraction_strategy). If there are no attributes and the facet_field's attribute_key is "category_ids", the size is set to 500. Otherwise, the default size is set to 100.
The FacetClass is executed to retrieve the matching values, and the key-value pairs of facet_fields for products are obtained. Then, the inventory_strategy is returned since the facet_fields, such as price_list and inventory_strategy, vary. For all defined filtering types in the database, the "._search" method executes the facet_generator and continues the flow.
The retrieved facets from Elasticsearch, which represent the attributes (e.g., in_stock), are used to filter the products. The IDs of these attributes are used to fetch their corresponding values from the database, and widgets are created using these values.
omnishop.search.services.SearchService._search
Facet Field and Sort Options code blocks that retrieve can be seen below.
The data type responsible for providing sorting on the product list page in the shop is the sort_options data type. This data type is stored in the database and managed by Omnitron for related operations.
pre_filter_kwargs and pre_exclude_kwargs are parameters that can be sent to add dynamic rules if desired. They are managed through settings.
In the settings, under Dynamic Settings, you can add the following examples for SEARCH_PRE_FILTER_KWARGS and SEARCH_DYNAMIC_FILTER:
Kwargs Dynamic Filters:
Search Pre Filter Kwargs Dynamic Settings:
The pre_filter_kwargs and pre_exclude_kwargs values are utilized in the omnishop.search.services.SearchService._search method of the SearchService.
Pre Filter and Exclude Kwargs:
To dynamically add rules to the pre_filter_kwargs value mentioned above, as indicated, this value is obtained through dynamic_settings. For this purpose, the "SEARCH_PRE_FILTER_KWARGS" value is retrieved via the "_get_pre_filter_kwargs()" method.
Relevant code block can be seen below:
Getting Pre Fılter Kwargs From Dynamic Settings:
min_score: When querying products in Elasticsearch, those with scores below this value are excluded. This value can be managed with settings.
Es Min Score Coefficient:
After receiving a response from Elasticsearch with the information above, the product extraction process takes place. For this purpose, there are several ProductExtractionStrategies available. If the extraction strategy is not specified in the request, the DefaultProductExtractionStrategy is used.
Strategies:
DefaultProductExtractionStrategy: Retrieves all product_ids that match the query.
FirstItemSearchExtractionStrategy: Retrieves the first item from the list of sub-products.
ListableFirstProductExtractionStrategy: Retrieves listable products that are included in the meta product's sub-products.
MinPriceFirstProductExtractionStrategy: Sorts the sub-products based on the minimum product price in ascending order (a-z) and retrieves the first product.
MaxPriceFirstProductExtractionStrategy: Sorts the sub-products based on the maximum product price in ascending order (a-z) and retrieves the first product.
AllSubProductsExtractionStrategy: Retrieves all sub-product ids belonging to the meta product.
After obtaining the facets and product IDs from Elasticsearch, conversion processes are performed as follows:
Product Convert: The product_ids received from Elasticsearch are used to retrieve products from the database, maintaining their original order.
Facet Convert: The facets (attribute filters) received from Elasticsearch are used to retrieve attribute values from the database, which can be used to filter the products. Widgets are created based on these attribute values. At this stage, different widgets are created based on the attribute keys.
If a Facet Configuration is specified for a relevant attribute key, a widget is created accordingly. For category_ids, a Category Widget is created. For basket_offer_ids, a Promotion Widget is created. For the Attribute Value Widget, attribute keys should be provided in the settings.
For category_ids, create a Category Widget.
For basket_offer_ids, create a Promotion Widget.
For the Attribute Value Widget, attribute keys should be provided in the settings.
Landing Page Attributes:
If none of the above conditions are met, the attribute key will default to creating a Multi Select Widget. After the conversion process, the response is generated. The response includes the following fields:
Fields:
facets
sorters
search_text
products
Dynamic Settings Used Under Search
The dynamic settings configurations that exist beneath the search structure on the Shop side and are utilized in the workflow are summarized below, along with their purposes.
1. HIDE_GROUPED_PRODUCTS_ON_SEARCH
Why is it used?
It is used to remove combination product listings from the search structure.
How does it work?
When this setting is enabled, during indexing, the "name" and "search" fields of the Elasticsearch product object are set as empty strings. This means that when group products are sent to Elasticsearch, the "name" and "search" fields are sent as empty strings when this setting is enabled.
2. PRODUCT_STOCK_OUT_VISIBILITY_ENABLED
Why is it used?
It is used to display out-of-stock products as "sold out" on the listing page.
How does it work?
The setting "PRODUCT_STOCK_OUT_VISIBILITY_ENABLE" is set to True in the dynamic settings. This enables the functionality to display products as "sold out" when they are out of stock on the listing page.
3. SEARCH_DEFAULT_PAGE_SIZE
Why is it used?
It is used to dynamically change the page size on the product listing page. For example, instead of loading a new page with 20 products, it can be changed to load a new page with 24 products, allowing for flexibility in the number of products displayed.
How does it work?
The dynamic setting "SEARCH_DEFAULT_PAGE_SIZE" is used with an integer value desired for the page size, along with a minimum value also provided as an integer. This setting allows for the customization of the page size on the product listing page.
Settings Edited via Omnitron
Sort Options
Under the section Requesting Elasticsearch, it was mentioned that sort options are the data type responsible for sorting on the product listing page in the shop. The addition, modification, or removal of these options is controlled through Omnitron. The steps to perform these actions are as follows:
Access the Settings section in the Omnitron panel.
Click on the Sorting Algorithms field in the opened screen.
On the opened screen, you can see the sorting configurations for the relevant catalogs under the headings of names, sorting types, and visibility.
By clicking on the relevant sorting algorithm, you can make edits, or you can add a new sorting algorithm using the New Sorting Algorithms field located in the top right corner.
AutoComplete
The autocomplete system automatically starts working when a character is entered in the search field. It takes the entered search_text as a parameter and begins executing within the autocomplete function. If there is no search_text provided when this service runs, the suggested results will return as an empty list. In this section, the products stored in Elasticsearch are queried. Multiple matches are aimed to be achieved by using different types in these queries. Additionally, the ES_MIN_SCORE_COEFFICIENT value is included when querying the products. If this value is greater than zero, the min_score value received from here is added, and the query is updated accordingly.
The obtained search object is then modified based on the configurations defined in the sort and pre_filter functions, respectively. As a result of these sorting operations, the four most suitable results are returned based on the search_text. These four results represent product_ids. These products are automatically listed on the search label.
Additionally, a listing process is performed for categories as well. The products are checked, and if they have categories and there are relevant results, they are also listed and shown at the top of the search results.
Sent Requests
Search
Index Configuration
IndexConfiguration is a model that enables the addition of custom values to Elasticsearch. By creating an IndexConfiguration, numeric attribute values can be indexed in Elasticsearch, allowing for their later use for various purposes.
There are a few points to pay attention to here. First, we expect the attribute value to be numeric or convertible to a numeric value. For example, values like 1, 100, or "50" are acceptable at this stage. However, if our attribute values contain different formats, such as "30 cm" or "10x20x50," non-numeric values will not be indexed. To create an IndexConfiguration object, a sample request should look like this:
POST https://{storefront_url}/api/v1/index-configurations/
POST https://{storefront_url}/api/v1/index-configurations/Request
All three fields must be included in the body. The 'name' field should correspond to an existing 'attribute_key'; you cannot pre-add it for a non-existent attribute in the system. The 'attribute_type' value can be integer, float, or long. If a request is made for a non-existent 'attribute_key,' the response will be as follows:
Response
Assuming a successful request with an existing 'attribute_key,' the response will be as follows:
Response
Compatibility of IndexConfigurations with Facets
After creating IndexConfigurations, the desired data will be indexed in Elasticsearch. The way data is used after this stage is up to the user. To illustrate how custom usage can work, let's assume we have successfully created an IndexConfiguration object as described above, and there are products associated with the attribute_key used in this object.
In order to create a rangable facet, you can create a new FacetConfiguration object. For example, after creating the IndexConfiguration object and indexing data into Elasticsearch, the key used for indexing is formed as "custom_" + indexconfiguration.attribute_key. For example, if our attribute_key in the defined object is 'exist_attribute_name,' the indexed value in Elasticsearch will be 'custom_exist_attribute_name.'
To ensure compatibility, our FacetConfiguration object should also have an attribute_key value of 'custom_exist_attribute_name.' This will allow us to create a facet for the indexed custom data.
Let's consider that our IndexConfiguration object is created with the following values:
The values for the sample facet object we need to create should be defined as follows:
POST https://{storefront_url}/api/v1/facet/
POST https://{storefront_url}/api/v1/facet/Request
With this created object, you can now use this filter for products with the indexed custom_integration_cap value on the listing page.
Last updated
Was this helpful?
