Table of Contents

Creating indexes

How to create new indexes in DW10

DynamicWeb uses Lucene.NET - a high performance and full-featured text search engine library - to implement search capabilities in frontend and backend.

Lucene.NET works by creating indexes - data structures which store information about where in the database something can be found. Think of an index as a specialized and very fast database for finding documents which contain a specific text string, for the technical details checkout the How Lucene.NET works article.

DynamicWeb 10 contains standard components for creating the following types of indexes:

Each of these is used to index a particular type of content on a solution, except for the SQL index which is a flexible option when you need to create non-standard indexes. This won’t be necessary very often – most solutions will need a Product-index, a Content-index, and maybe a User-index. But it’s there for those edge-cases that always pop up.

To create an index navigate to a repository, then click manage indexes and add a new one: Creating an index

An index contains the following elements:

  • Instances – the index files being searched when a query is executed
  • Builds – instructions detailing exactly how you want the index to be built
  • Fields – holds or points to indexed and possibly analyzed content you can query

Click in the newly created index to open the index overview screen where we will work with these elements: Index overview screen

Instances

An instance is a set of index-files located on disk, the files you query when you search. The instance files are saved to disk and can be found under /Files/System/Indexes/[RepositoryName]/[IndexName]/[InstanceName] on the server or via the Assets area.

Instances are created using the Action-menu... CreatingInstances ...and you typically want to create two instances, so that one is always active as the other one is being built - which leads us to...

Builds

Builds are instructions to the system about the kind of index you're building and what to include or exclude from the index. Each type of index - Products, Users, Content, etc. - has a specific builder associated with it.

In this example I am adding a build configuration for creating a File index: Adding a build configuration As you can see, the only builder action available is Full and I am sticking to the default builder settings on this occasion.

You can find information about each builder and the builder settings available to you in the documentation for each index type linked to in the introduction to this article.

Fields

Lucene-indexes are structured as collections of small documents divided into fields, which can either hold content that's searchable or point to data that can be fetched later. So when creating an index you need to tell the builder which fields you want to include in the index.

In most cases your point of departure will be a set of predefined field collections added via an add-in called a SchemaExtender. We deliver the following SchemaExtenders out of the box:

  • The ProductIndexSchemaExtender adds product fields to an index
  • The ContentIndexSchemaExtender adds page and item fields to an index
  • The UserIndexSchemaExtender adds user fields and a set of calculated user-related fields to an index
  • The FileIndexSchemaExtender adds file-related fields to an index

To add fields via a SchemaExtender you add a field of the type SchemaExtender and select the one you want to use, in this example the FileIndexSchemaExtender: Adding SchemaExtender fields After saving, all fields from the schema extender are added to the index and available to query: SchemaExtenderFields

Since fields from a SchemaExtender are pre-configured, we've made some decisions about how they're indexed - specifically whether they're stored, indexed and/or analyzed:

  • Stored fields have their values stored in the index
  • Indexed fields can be searched, the value is stored as a single value unless analyzed
  • Analyzed fields have their values run through an analyzer and split into tokens before being indexed

You can read more about e.g. analyzers, tokenizers, and the tehnical details of indexing in the How Lucene.NET works article. For most practical purposes it's sufficient to know that:

  • A field you want to query or display in frontend should be indexed
  • If you want to search for parts of the text it should be analyzed
  • If you want to use the complete value as e.g. a label in a facet it should not be analyzed
  • Fields which should be published using the query publisher app should be stored

If the default-configuration available via the SchemaExtender does not fit your use-case you can of course always add fields manually and have full control over how they are indexed.

Manually adding fields

To add fields to an index manually repeat the process from when you added a schema extender field - but choose one of the other field types. You have access to the following fields:

  • Simple fields
  • Summary fields
  • Grouping fields

A simple field is an index field which maps e.g. a single product attribute like name. You typically add a simple field to an index because the field from the SchemaExtender is analyzed, and you need the unanalyzed value for facets.

To add a simple field to an index first open the field overview and then:

  1. Click New index field and select Field
  2. Provide a name and a system name
  3. In the Field-section:
    • Select a field type matching the data you're indexing
    • Optionally set a boost value to rank search results matching this field higher than default
    • Check Stored, Indexed, and Analyzed as appropriate
  4. Select a source field
  5. Save

You can select any source field made available by the index builder - typically standard fields - as well as custom sources made available by a builder extender.

Custom field types

Occasionally you may have some custom data that you want to index in a specific way - one which doesn't work with the standard field types which all use the Lucene StandardAnalyzed. In those cases you can create a custom field type and apply a different analyzer.

To create custom field type using a different analyzer:

  1. On the index overview screen use the context menu for the Index fields-widget to select Manage field types
  2. Click New field type
  3. Provide a name and select a data type
  4. Optionally set a boost value to rank search results matching this field higher than default
  5. Select the analyzer you want to use
  6. Save

Once created, custom field types can be selected in the normal way when you add a field to an index.

Tip

The Lucene StandardAnalyzer uses a built-in list of stop words which will not be indexed and cannot be searched for. The default list of stop words contains the following:

a
an
and
are
as
at
be
but
by
for
if
in
into
is
it
no
not
of
on
or
such
that
the
their
then
there
these
they
this
to
was
will
with

You can override the default list by creating a file called stopwords.txt in the /Files/System/Repositories folder – the format must be 1 stop word per line.

Queries and Facets

On the index overview screen you also have widgets for queries and facets - but as they are not part of the index creation process they are not covered in this article.

Improve this doc
To top