Table of Contents

Setting up a Product Search From Scratch with Facets

This guide will walk you through the process of setting up a product search with facets from scratch.

What You'll Learn

We'll cover the following topics:

  1. Creating a Repository: Establish a new repository to organize your product data
  2. Creating an Index: Set up an index within the repository to structure and manage your data effectively
  3. Adding Fields: Define fields in the index and extract relevant information from your source data
  4. Creating a Search Query: Develop a search query to retrieve data based on user input
  5. Setting Up Facets: Implement simple facets to enhance search functionality and improve the user experience

This guide will not delve into every detail of how each of these components works. Instead, it aims to provide a holistic overview of how they all work together. By the end, you’ll have a foundational understanding that enables you to explore and master more complex configurations on your own.

Let's dive into the step-by-step process of creating a product search with facets in DynamicWeb, starting with setting up your repository.

Creating a Repository

A repository is a centralized storage location where product data is managed and organized. By creating a repository, you establish a structured environment that supports efficient data retrieval and management. Think of it as kind of a top folder for indexing configuration files – each repository can contain a number of indexes, queries etc.

To create a repository:

  1. Navigate to Settings and click the 'Plus-symbol' next to Repositories
  2. Provide a name
    • Since we're setting up a repository for our products, we'll call the repository Products

This creates a new repository and opens the configuration page. Note that it contains 4 subsections:

  1. Indexes
  2. Queries
  3. Facets
  4. Tasks

Repo_01

Creating an Index

We'll start by creating an Index.

Indexes are structures within the repository that organize and store your content - in the case of a product index, this will be products. By indexing your data, you ensure that search queries can be executed quickly and accurately, providing relevant results to users.

This is a fairly complicated process, but of course it only needs to be done once, after which the index can be automatically rebuilt at intervals.

Creating a working index requires you to add the elements in the proper order: Index > Instance > Build > Fields > Building the Index.

To create an Index:

  1. In the indexes section, click Manage
  2. Click New index
  3. Provide a name
  4. Select a Balancer
    • Dynamicweb.Indexing.Balancing.ActivePassive: Selects the next instance on the list of instances – so if instance A is unavailable (building, has failed), instance B will be used unless it’s unavailable, in which case instance C will be used, and so on
    • Dynamicweb.Indexing.Balancing.LastUpdated: Directs operations to the most recently updated index, ensuring users interact with the freshest data
  5. Click Save and Close

Notice how your newly created index appears in the indexes section in your repository. However, at this point, the index is just an empty shell. To make it functional, we need to build what's inside the index. We'll start with:

Instances

An instance refers to a specific indexed file stored in the file archive. This file contains the structured and optimized data used for executing search queries. When a user performs a search, the system searches through this instance to retrieve relevant results quickly and efficiently. It is usually recommended that you create at least two intances - that way users will still be able to search while the other instance is being (re)built.

To create two instances:

  1. Enter your newly created Index
  2. Click the Actions button on the top right corner and select Manage instances
  3. Click New instance
  4. Provide a name, e.g. 'A' for the first instance and 'B' for the second
  5. Select the LuceneIndexProvider
  6. Specify a folder to place the instance file under
  7. Click Save and close
  8. Repeat the process for the second instance

Notice how your newly created instances appear in your index: search_02

Building the Instance

Now that we have two instances, the next step is to create a build configuration for them:

For a product index, the associated builder is called the ProductIndexBuilder. This builder is designed to handle the unique requirements of product data, ensuring that all relevant attributes (such as product names, descriptions, prices, categories, etc.) are indexed correctly.

To create a build configuration:

  1. In your Index, under the Builds section, click Manage
  2. Click New build
  3. Provide a name, e.g. 'ProductIndexBuildDefinition'
  4. In the Builder section, select Dynamicweb.Ecommerce.Indexing.ProductIndexBuilder
  5. This opens a selection of builder settings
  6. Select a Builder action:
    • Since this is the initial build of our Instance, we'll select 'Full'
  7. Scroll down and check the 'Handle inherited category values' checkbox
  8. Review the rest of the build settings
    • As you can see, there are many options available, and we won't go into detail in this tutorial. For now, just leave them as they are. If you want to learn more, you can check out the documentation
  9. Click Save and close

Head back to your index and note that a 'build-button' has been added below your instances: search_03

However, don't click this build button just yet. You’ve only specified how you want the index to be built – next, you should specify what you want to include in the index.

Adding Fields to the Index

Before we add any fields to our index, let's go over what a field actually is. Fields are the fundamental units that store data attributes for documents within an index. Each field represents a specific piece of information about the documents. In the case of a product index, this could be the product's name, price, or description.

To understand what a field is, let's look at a simple product catalog:

ProductID ProductName Description Price Category
1 Smartphone Latest model 299.99 Electronics
2 Laptop High performance 999.99 Computers
3 Headphones Noise-cancelling 199.99 Accessories

From this simple example, we can understand some key components of fields:

  1. Fields: 'ProductID', 'ProductName', 'Description', 'Price', and 'Category' are the fields.
    • Values: Each of these fields can have multiple values. For example, the 'Price' field can have the values '299.99', '999.99', etc.
      • Type: Each of these values is a specific data type e.g. numerical (299.99) or string (Latest model)

Stored, Indexed, and Analyzed Fields

Furthermore, fields can be either stored, indexed, or analyzed. This is where it gets a bit complicated and we wont go into every detail. If it has your interest, you can refer to our documentation for how indexing works here and analyzers work here.

Basically, it works like this:

  1. Stored Fields
    • Definition: Stored fields have their values stored in the index exactly as they are for later retrieval using the e.g. the query publisher. Usually it is not needed to store any field in the index as the values are not used by the product catalog app - except ProductAutoID. The product catalog app will use the autoid to retrieve the product from the database.
    • Use Case: When you need to retrieve and display the original value from the index using the query publisher
    • Example: Storing the product description
      • Field: 'Description'
      • Value: "Latest model"
      • Purpose: When a search result is displayed using the query publisher, the exact description can be shown to the user
  2. Indexed Fields
    • Definition: Indexed fields are searchable and stored in an index as single, unprocessed values. These fields are typically used for exact match searches, and the value is not modified or tokenized during indexing unless explicitly set to be analyzed.
    • Use Case: When you need to search for exact matches of a value
    • Example: Indexing the product ID
      • Field: 'ProductID'
      • Value: "2"
      • Purpose: Allows for fast lookup of a product by its unique identifier
  3. Analyzed Fields
    • Definition: Analyzed fields have their values run through an analyzer, which splits the value into tokens (words) for full-text search. For example, the description "High performace" will be split into the tokens (words) "high" and "performance"
    • Use Case: When you need to support full-text search and find matches based on partial or full terms
    • Example: Analyzing the product description
      • Field: 'Description'
      • Value: "Latest model with advanced features"
      • Purpose: Allows searching for individual terms like "latest", "model", "advanced", or "features"
Note

The key to understanding how fields work in an index lies in recognizing the roles of stored, indexed, and analyzed fields, and how they interact with different data types (String, Numerical, etc.). Once you grasp these concepts, you can effectively create your indexes to support different search and retrieval needs.

Adding SchemaExtenders

Defining each field that a product index can contain requires a lot of work. Therefore, DynamicWeb comes with a set of predefined SchemaExtenders for standard index types such as product, content, user, and file. These SchemaExtenders add a bunch of predefined fields to the indexes, covering most of the needs for a standard index.

To add Schema Extenders:

  1. In YourIndex in the Index Fields section, click the three dots, and select Manage Fields
  2. Click New index field and select Schema extender
  3. Provide a name and a system name
  4. Click the Type dropdown, you'll see two options:
    • ConfigurableProductIndexSchemaExtender
    • ProductIndexSchemaExtender
    • Note: The regular ProductIndexSchemaExtender automatically adds many commonly used fields, making setup easier. However, for this tutorial, we’re doing things from scratch, so choose the ConfigurableProductIndexSchemaExtender, which adds only the essential fields, allowing you to define the rest yourself.
  5. Click Save and close

Notice that a bunch of fields has been added to your index - these are fields in the Schema Extender have preconfigured for you.

Adding Simple Fields

The Configurable Product Index Schema Extender only adds a few essential fields to cover basic functionality. However, to enable product search and facets, we'll need to create and define our own fields from scratch.

We'll add two types of fields:

  1. Search fields: Fields used primarily for searching. These fields are analyzed to support full-text search capabilities
    • Example:
      • Text Search: Supports searching within longer text fields, such as product descriptions. For example, searching for "high resolution" in the 'Description' field will match products containing that phrase
  2. Facet fields: Fields used for filtering search results. These fields should not be analyzed to ensure accurate facet filtering
    • Example:
      • Color Filtering: Allows users to filter products by color. For example, selecting "Black" in the 'Color' facet will show only black products
Creating the Search Fields

Let's start with the search fields:

We'll create these search fields:

  • Product Name
  • Short Description

Let's start with the Product Name field.

To create this field:

  1. Click New index field and select Field
  2. Provide a name, e.g. 'ProductName'
  3. Provide a system name, e.g. 'ProductName'
  4. Select a data type
    • Select 'System.String', since the name of a product is textual data
  5. Set the boost value to 5 since the name of the product is generally of relatively large importance compared to the other fields
  6. Choose whether the field should be stored, indexed, or analyzed. Here, we'll:
    • Leave Stored unchecked. Storing is unnecessary for fields that's used for searching
    • Check Indexed: The fields needs to be included in a searchable 'inverted index'
    • Check Analyzed: This enables the search for the product name to be flexible
      • Example: A Product named "Smart Speaker" will be divided into two tokens: "smart" and "speaker", enabling the user to search for both 'smart' and 'speaker' to find this product
  7. In the Settings section, you'll need to select a source. This maps the data that will be in your field to the source data
    • What you should choose here depends on the source data available to you. In the standard swift products catalog, it's called 'ProductName'
  8. Click Save and close

Next up is the Short Description field. Creating this is very similar to the Product Name field:

  1. Click New index field and select Field
  2. Provide a name, e.g. 'ShortDescription'
  3. Provide a system name, e.g. 'ShortDescription'
  4. Select the 'System.String' as the data type
  5. Leave the boost value empty
  6. Choose whether the field should be stored, indexed, or analyzed. Here, we'll:
    • Leave the Stored unchecked
    • Check Indexed
    • Check Analyzed
  7. In the Settings section, select 'ProductShortDescription' as the source
  8. Click Save and close
Creating the Facet Fields

Let's move on to the Facet fields.

We'll create these facet fields:

  • Color
  • Price Range

Let's start with the Color field.

To create this field:

  1. Click New index field and select Field
  2. Provide a name, e.g. 'Color_Facet'
  3. Provide a system name, e.g. 'Color_Facet'
  4. Select a data type
    • Select 'System.String[]', since the color of a product is textual data
  5. Leave the boost value empty
  6. Choose whether the field should be stored, indexed, or analyzed. Here, we'll:
    • Check Stored: This allows the value to be displayed in the user interface
    • Check Indexed: The field needs to be included in a searchable 'inverted index'
    • Leave Analyzed unchecked: Analyzing can distort the displayed facets
      • Example: The colour facet 'Light Blue' will be split into two facets 'light' and 'blue'
  7. In the Settings section, you'll need to select a source. This maps the data that will be in your field to the source data
    • What you should choose here depends on the source data available to you. In the standard swift products catalog, it's called 'ProductCategory│Reference_fields│Color'
  8. Click Save and close

Next up, let's create the Price Range field. Again this is very similar to the color field, but with a few key differences:

  1. Click New index field and select Grouping Field
  2. Provide a name, e.g. 'PriceRange_Facet'
  3. Provide a system name, e.g. 'PriceRange_Facet'
  4. Select a data type
    • Select 'System.String', since the price range should be a string and not numerical to allow for user friendly display, e.g. "$0 - $50" or "Above $200"
  5. Leave the boost value empty
  6. Choose whether the field should be stored, indexed, or analyzed. Here, we'll:
    • Check Stored
    • Check Indexed
    • Leave Analyzed unchecked
  7. In the Settings section, select a ProductPrice
  8. In the Groups section, you can create a group (e.g. 1 - 200) like this:
    • Click Add
    • Name: '1 - 200'
    • From: 1
    • To: 200
      • Create as many groups as you want, in the ranges you want.
  9. Click Save and close

That's it! You've created a mini product index from scratch! Learning to create custom fields is essential for handling unique cases that SchemaExtenders might not address.

Now, head back and click the Build button beneath your instance: build_01

Now that we've set up the index, defined the fields, and built our instance, the next step is to create queries.

Creating a Query

Queries are essential for retrieving the data from the indexes and fields we've created. They allow users to search and filter product data effectively. Let's set up a search query.

To create a simple query:

  1. Navigate to YourRepository
  2. In the Queries section, click Manage
  3. Click New Query
  4. Provide a name for the query
  5. Select 'YourIndex' as the Source index
  6. Click Save and close. This creates an empty query that you can now configure further

Enter the query and see that it contains 3 subsections:

  1. Parameters
  2. Expressions
  3. Sorting

Query_01

Expressions

The core of a query lies in the set of expressions which dictate the result it returns. You can think of an unmodified or 'empty' query as one that returns everything in the index. Each expression you add acts as a filter, narrowing down the results to match your specific needs.

In this tutorial, we'll create some really simple expressions that we can use to make a basic product name and short description search, and a simple color and price range facet. This requires that we do three things:

  1. Create a group for the expressions
  2. Create parameters used in the expressions
  3. Create the expressions

Creating Groups for Expressions

Start by creating a group. Groups are logical containers that organize expressions within a query. They determine how the expressions interact to filter results.

Types of Groups
  1. AND-Groups
    • Description: All conditions in the group must be met for a product to be included in the results
    • Example: If you have an AND-Group with conditions 'color = red' and price 'range = 0 - $200', only products that are red and priced between $0 and $200 will be shown
  2. OR-Groups
    • Description: At least one condition in the group must be met for a product to be included in the results
    • Example: If you have an OR-Group with conditions 'color = red' or 'price range = 0 - $200', products that are either red or priced between $0 and $200 will be shown

Additionally, you can select the Negate option for a group. This will return all products that do not match the expressions in the group.

Group for Product Name and Short Description Expressions

Let's start by creating the group for the product name and short description expressions:

  1. In YourQuery in the Expressions section, click Manage
  2. Click New Group and check the Or option
  3. Click Save
  4. See that an Or-Group has appeared below the root expression

In this group, we'll create two expressions:

  1. Product Name Search
  2. Short Description Search

We'll start by creating the Product Name Search expression:

Creating Parameters

However, before we can create this expression, we need to create a parameter. A parameter is a placeholder that captures user input dynamically, allowing queries to adapt based on the input provided. It makes the search more flexible and user-specific.

  1. Navigate to YourQuery
  2. In the Parameters section, click Manage
  3. Click Add under Parameters
  4. Provide the parameter with a name, e.g. 'Search'
  5. Select 'System.String' as the Type
  6. Leave the Default value empty, since the user will provide the input when they make a search for the products name
  7. Click Ok and click Save and close

Creating Expressions

Now we can create our expression for the Product Name Search

  1. Click the three vertical dots to the right of the Or-Group
  2. Click New expression
  3. In the Field dropdown menu, select the ProductName_Search field that you created in your index
  4. In the Operators dropdown menu, select Contains. The 'Contains' operator checks if the value in the field contains the value provided by the user
  5. In the Type dropdown menu, select Parameter and choose the parameter you've just created
  6. Click Save

For the Short Description expression, just follow the same steps, but select the ShortDescription field and reuse the Search. parameter.

Understanding the Logic Behind the Expression

Let's break down what we just did:

  • Parameter Creation: We created a parameter to capture the user's search input. For instance, if a user types "T-shirt", this input is stored in the parameter
  • Expression Setup: The expression uses the 'Contains' operator to compare the user's input with the values in the 'ProductName' or 'ShortDescription' fields. If one of the fields contains something that matches the input (e.g. "T-shirt"), that product will appear in the search results
  • Or-Group: We placed this in an OR-group to allow matching across multiple fields, such as product name or description
Color and Price Range Expressions

Next, is the Color and Price Range facet. This is very similar to the previous, but with a few key differences:

Start by making a Group:

  1. In YourQuery in the Expressions section, click Manage
  2. Click New Group and check the AND option
  3. Click Save
  4. An AND-Group has appeared below the root expression

Now we can add the expressions for the Color and PriceRange facets.

We'll start with the expression for the Color facet:

Again, start by creating a parameter:

  1. Enter YourQuery
  2. In the Parameters section, click Manage
  3. Click Add under Parameters
  4. Provide the parameter with a name, e.g. 'color_facet'
  5. Select 'System.String[]' as the Type
  6. Leave the Default value empty, since the user will provide the input when they select a facet
  7. Click Ok and click Save and close

After this, create the expression:

  1. In the AND-group, click the three vertical dots and click New expression
  2. In the Field dropdown menu, select the Color_Facet field that you created in your index
  3. In the Operators dropdown menu, select In
  4. In the Type dropdown menu, select Parameter and choose the 'color_facet' parameter you've just created
  5. Click Save

Next up is the Price Range facet.

  1. Create a parameter in the same way as before, but call it e.g. 'PriceRange_facet'
  2. Create a new expression in the same way as before, but select the PriceRange field and the newly created PriceRange parameter

Creating Facets

Next up, we'll create the Facets that'll use our facet fields combined with our facet queries to display the facets to the users in frontend.

First, create a Facet Group:

  1. Navigate to the content tree
  2. Fold out 'YourRepository' > 'YourIndex' until you reach 'YourQuery'
  3. Click the three vertical dots and click Add facet group
  4. Provide a name, since this is facets for displaying product information, call it e.g. 'ProductFacets'
  5. Select 'YourQuery' as the Query
  6. Click Save and close

Note that a facet group has appeared below your index in the content tree.

Adding Facets to the Facet Group

Next up, we'll add facets to this facet group.

We'll start with the Color facet:

  1. Enter 'YourFacetGroup' and click Add new facet
  2. Provide a name, e.g. 'Colors'
  3. Select Field under Facet Types
  4. Select your newly created color facet field under Field
  5. Select your newly created color facet query under Query parameter
  6. Select e.g. Colors in Render Type
  7. Select AlwaysVisible under Show
  8. Click Save

Next up is the Price Range facet:

  1. Click Add new facet
  2. Provide a name, e.g. 'Price Range'
  3. Select a Field under Facet Types
  4. Select your newly created price range facet field under Field
  5. Select your newly created price range query under Query parameter
  6. Select Checkboxes in Render Type
  7. Select AlwaysVisible under Show
  8. Click Save

And that's it! You've set up a product search with facets from scratch. With this foundational knowledge, you can explore more advanced configurations and tailor product searches to your specific needs.

Improve this doc
To top