[id='indexing-configuration_{context}']
= Index configuration
{brandname} configuration controls how indexes are stored and constructed.

[discrete]
== Index storage

You can configure how {brandname} stores indexes:

* On the host file system, which is the default and persists indexes between restarts.
* In JVM heap memory, which means that indexes do not survive restarts. +
You should store indexes in JVM heap memory only for small datasets.

.File system
[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_file_system.xml[]
----

.JVM heap memory
[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_jvm_heap.xml[]
----

[discrete]
=== Index path

Specifies a filesystem path for the index when storage is 'filesystem'.
The value can be a relative or absolute path. Relative paths are created
relative to the configured global persistent location, or to the current
working directory when global state is disabled.

By default, the cache name is used as a relative path for index path.

[IMPORTANT]
====
When setting a custom value, ensure that there are no conflicts between caches using the same indexed entities.
====

[discrete]
[id='indexing-configuration_startup-mode']
== Index startup mode

When {brandname} starts caches it can perform operations to ensure the index is consistent with data in the cache.
By default, it:

* Checks the existing index file format.
** If it is incompatible or corrupt, it is deleted and the cache is automatically reindexed.
* Automatically clear (purge) or reindex the cache.
** If data is volatile and the index is persistent then {brandname} performs the clear (purge) the indexes when it starts.
** If data is persistent and the index is volatile then {brandname} reindex the cache when it starts.

[NOTE]
====
The purge operation is performed synchronously, since it is usually very fast. So by the time the cache finishes to start, the operation will be completed.
The cache becomes available only when the purge completes.

The reindex operation is performed asynchronously, since it might take a longer time to complete, depending on the size of the cache.
If an indexed query is performed during the reindex the result could be partial.
It is always possible to check if a reindex is ongoing accessing to link:{query_docs}#getting-query-statistics_query-monitoring-tuning[the query statistics].
====

But you can manually configure it to:

* Purge the index when the cache starts.
* Reindex the cache when it starts.
* No indexing operation takes place when a cache starts

[NOTE]
====
In the case of a manual configuration can lead to possible inconsistencies,
a log message will be presented when the cache starts.
====

.Clear the index when the cache starts
[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_startup_purge.xml[]
----

.Rebuild the index when the cachin this case
a warning message will be logged when the cache is startede starts
[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_startup_reindex.xml[]
----

== Automatic strategy and shared cache stores

In case of:

1. A shared storage is configured
2. The indexes are persistent on file system

The `AUTO` startup mode will apply the `REINDEX` strategy.
This is done in order to not miss potential updates in case of crash and recovery of an indexed node.

Thus `AUTO` in this case may penalize the average user using a shared cache store that does not do any eviction,
since more reindex than necessary will be triggered.
For this reason if:

1. A shared storage is configured
2. The indexes are persistent on file system
3. Automatic evictions are not possible: cache memory `max-size` and `max-count` are not used
4. Manual eviction using the embedded API `cache.eviction()` is not used

to use `NONE` in place of `AUTO` as the index startup strategy.

[discrete]
== Indexing mode

`indexing-mode` controls how cache operations are propagated to the indexes.

`auto`:: {brandname} immediately applies any changes to the cache to the indexes. This is the default mode.

`manual`:: {brandname} updates indexes only when the reindex operation is explicitly invoked. Configure `manual` mode, for example, when you want to perform batch updates to the indexes.

Set the `indexing-mode` to `manual`:

[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_manual.xml[]
----

[discrete]
== Use Java Entities

If the cache is protostream-encoded and the indexes initialized from a {brandname} server instance,
the indexed entities must be the indexed Protobuf messages defined on some Proto schema.
It is possible to change this behavior forcing the indexes be defined on the indexed entities that are discovered
from the java entities locally accessible from the server VM.
Useful in case we want to run embedded queries from a server task, in the case the cache is Protobuf encoded.

[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_java_entities.xml[]
----

[discrete]
== Index reader

The index reader is an internal component that provides access to the indexes to perform queries. As the index content changes, {brandname} needs to refresh the reader so that search results are up to date.
You can configure the refresh interval for the index reader.
By default {brandname} reads the index before each query if the index changed since the last refresh.

[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_index_reader.xml[]
----

[discrete]
== Index writer

The index writer is an internal component that constructs an index composed of one or more segments (sub-indexes) that can be merged over time to improve performance.
Fewer segments usually means less overhead during a query because index reader operations need to take into account all segments.

{brandname} uses Apache Lucene internally and indexes entries in two tiers: memory and storage.
New entries go to the memory index first and then, when a flush happens, to the configured index storage.
Periodic commit operations occur that create segments from the previously flushed data and make all the index changes permanent.

[NOTE]
====
The `index-writer` configuration is optional.
The defaults should work for most cases and custom configurations should only be used to tune performance.
====

[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_index_writer.xml[]
----

.Index writer configuration attributes
[%header,cols=2*]
|===
|Attribute
|Description

|`commit-interval`
|Amount of time, in milliseconds, that index changes that are buffered in memory are flushed to the index storage and a commit is performed. Because operation is costly, small values should be avoided. The default is 1000 ms (1 second).

|`max-buffered-entries`
|Maximum number of entries that can be buffered in-memory before they are flushed to the index storage. Large values result in faster indexing but use more memory. When used in combination with the `ram-buffer-size` attribute, a flush occurs for whichever event happens first.

|`ram-buffer-size`
|Maximum amount of memory that can be used for buffering added entries and deletions before they are flushed to the index storage. Large values result in faster indexing but use more memory. For faster indexing performance you should set this attribute instead of `max-buffered-entries`. When used in combination with the `max-buffered-entries` attribute, a flush occurs for whichever event happens first.

|`thread-pool-size`
|This configuration is ignored since Infinispan 15.0. The indexing engine now uses the Infinispan thread pools.

|`queue-count`
|Default 4. Number of internal queues to use for each indexed type. Each queue holds a batch of modifications that is applied to the index and queues are processed in parallel. Increasing the number of queues will lead to an increase of indexing throughput, but only if the bottleneck is CPU.

|`queue-size`
|Default 4000. Maximum number of elements each queue can hold. Increasing the `queue-size` value increases the amount of memory that is used during indexing operations. Setting a value that is too small can lead to `CacheBackpressureFullException` or `RejectedExecutionExceptionOperationSubmitter` since index operation requests are never blocked. In this case to solve the issue increase the `queue-size` or set the `queue-count` to 1.

|`low-level-trace`
|Enables low-level trace information for indexing operations. Enabling this attribute substantially degrades performance. You should use this low-level tracing only as a last resource for troubleshooting.

|===

To configure how {brandname} merges index segments, you use the `index-merge` sub-element.

.Index merge configuration attributes
[%header,cols=2*]
|===
|Attribute
|Description

|`max-entries`
|Maximum number of entries that an index segment can have before merging. Segments with more than this number of entries are not merged. Smaller values perform better on frequently changing indexes, larger values provide better search performance if the index does not change often.

|`factor`
|Number of segments that are merged at once. With smaller values, merging happens more often, which uses more resources, but the total number of segments will be lower on average, increasing search performance. Larger values (greater than 10) are best for heavy writing scenarios.

|`min-size`
|Minimum target size of segments, in MB, for background merges. Segments smaller than this size are merged more aggressively. Setting a value that is too large might result in expensive merge operations, even though they are less frequent.

|`max-size`
|Maximum size of segments, in MB, for background merges. Segments larger than this size are never merged in the background. Settings this to a lower value helps reduce memory requirements and avoids some merging operations at the cost of optimal search speed. This attribute is ignored when forcefully merging an index and `max-forced-size` applies instead.

|`max-forced-size`
|Maximum size of segments, in MB, for forced merges and overrides the `max-size` attribute. Set this to the same value as `max-size` or lower. However setting the value too low degrades search performance because documents are deleted.

|`calibrate-by-deletes`
|Whether the number of deleted entries in an index should be taken into account when counting the entries in the segment. Setting `false` will lead to more frequent merges caused by `max-entries`, but will more aggressively merge segments with many deleted documents, improving query performance.

|===

[role="_additional-resources"]
.Additional resources
* link:../../configuration-schema/index.html[{brandname} configuration schema reference]

[discrete]
== Index sharding

When you have a large amount of data, you can configure {brandname} to split index data into multiple indexes called shards.
Enabling data distribution among shards improves performance.
By default, sharding is disabled.

Use the `shards` attribute to configure the number of indexes.
The number of shards must be greater then `1`.

[source,xml,options="nowrap",subs=attributes+]
----
include::xml/indexing_sharding.xml[]
----
