:page-aliases: transforms/embeddings.adoc
// Category: debezium-using
// Type: assembly
// ModuleID: embeddings-transformation
// Title: Embeddings Transformation
[id="embeddings-transformation"]
= Embeddings Transformation
ifdef::community[]
:toc:
:toc-placement: macro
:linkattrs:
:icons: font
:source-highlighter: highlight.js

toc::[]
endif::community[]

One important task that must be performed to prepare content for use in large language models, or for natural language processing in general, is to convert text to a numeric representation, also known as text vectorization.
One way to implement this conversion is through so-called _text embeddings_, a process in which text is converted to high-dimensional numerical vectors.
Representing text as a vector enables computers to perform semantic similarity searches and other advanced operations.

{prodname} offers a built-in feature to transform specified text fields into numerical embeddings vectors, and to add the resulting embeddings to the event record.
Embedding inference is performed according to the embeddings model served by the configured provider.
{prodname} offers several embeddings model providers.

{prodname} can use a link:{link-kafka-docs}/#connect_transforms[single message transformation] (SMT) to generate embeddings.
To interact with different embeddings providers, this Embeddings transformation uses the link:https://docs.langchain4j.dev/[langchain4j] framework.

[IMPORTANT]
====
Embeddings transformation requires Java 21 or higher.
====

== Behavior

The Embeddings transformation takes as input specific fields in the original event record, and passes the text content of those fields to the configured embedding model for inference. 
That is, the SMT creates embeddings from the text contained in the specified fields.
The resulting embeddings are appended to the record.
The original source field is also preserved in the record.

The source field must be a string field.
The value of an embedding field is a vector of floating-point 32-bit numbers.
The size of the vector depends on the selected model.
To provide the internal representation of an embedding, {prodname} uses the link:https://github.com/debezium/debezium/blob/main/debezium-core/src/main/java/io/debezium/data/vector/FloatVector.java[FloatVector] data type.
The schema type of the embedding field in the record is `io.debezium.data.FloatVector`.

Both source field and embedding field specifications support nested structures, such as, `after.product_description_embedding`.

== Configuration

To configure a connector to use the embeddings transformation, add the following lines to your connector configuration:

[source]
----
transforms=embeddings,
transforms.embeddings.type=io.debezium.ai.embeddings.FieldToEmbedding
----

You must specify at least one field to use as the input for the embedding.
Destination field where the embedding will be placed is not mandatory.
If it's not specified, message value will contain only the embedding itself.
However, it's recommended to specify the embedding destination field, for example:

[source]
----
transforms.embeddings.field.source=after.product
transforms.embeddings.field.embedding=after.product_embedding
----

Finally, you must place the model provider JAR file in the connector class path, and configure the provider.
For example, for the Ollama provider, add `debezium-ai-embeddings-ollama-$VERSION.jar` to your connector class path, and add the Ollama URL and model name to the connector configuration, as shown in the following example:

[source]
----
transforms.embeddings.ollama.url=http://localhost:11434
transforms.embeddings.ollama.model.name=all-minilm
----

The following example shows what the full configuration for an Ollama provider might look like:

[source]
----
transforms=embeddings,
transforms.embeddings.type=io.debezium.ai.embeddings.FieldToEmbedding
transforms.embeddings.field.source=after.product
transforms.embeddings.field.embedding=after.product_embedding
transforms.embeddings.ollama.url=http://localhost:11434
transforms.embeddings.ollama.model.name=all-minilm
----

=== General configuration options

.Descriptions of embedding SMT configuration options
[cols="30%a,25%a,45%a",subs="+attributes",options="header"]
|===
|Option
|Default
|Description

|[[embeddings-source-field]]xref:embeddings-source-field[`embeddings.field.source`]
|No default value
|Specifies the field in the source record to use as an input for the embeddings.
The data type of the specified field must be `string`.
|[[embeddings-embedding-field]]xref:embeddings-embedding-field[`embeddings.field.embedding`]
|No default value
|Specifies the name of the field that the SMT adds to the record to contain the text embedding.
If no value is specified, the resulting record contains only the embedding value.
|===

== Model provider configuration

=== Hugging Face

Embeddings provided by models available via link:https://huggingface.co//[Hugging Face].

.Configuration options for Hugging Face embeddings
[cols="30%a,25%a,45%a",subs="+attributes",options="header"]
|===
|Option
|Default
|Description

|[[embeddings-huggingface-access-token]]xref:embeddings-huggingface-access-token[`embeddings.huggingface.access.token`]
|No default value
|Hugging Face access token.
|[[embeddings-huggingface-model-name]]xref:embeddings-huggingface-model-name[`embeddings.huggingface.model.name`]
|No default value
|Name of the embedding model.
Use the REST API to retrieve the list of available models.
Specify the provider in the REST call, for example, link:https://huggingface.co/api/models?inference_provider=hf-inference[Hugging Face inference provider].
|[[embeddings-huggingface-baseurl]]xref:embeddings-huggingface-baseurl[`embeddings.huggingface.baseUrl`]
|https://api-inference.huggingface.co/
|The base Hugging Face inference API URL.
|[[embeddings-huggingface-operation-timeout-ms]]xref:embeddings-huggingface-operation-timeout-ms[`embeddings.huggingface.operation.timeout.ms`]
|15000 (15 seconds)
|Maximum amount of time in milliseconds to wait for the embeddings reply.
|===

[NOTE]
====
Hugging Face started to support different link:https://huggingface.co/blog/inference-providers[embedding inference providers], including link:https://huggingface.co/docs/inference-providers/en/index[external providers].
However, the LangChain4j framework does not yet support external providers.
As a result, the only inference provider currently available for use with the {prodname} is the Hugging Face provider.
====


=== Ollama

Supports any model provided by link:https://ollama.com/[Ollama] server.

.Ollama embeddings configuration options
[cols="30%a,25%a,45%a",subs="+attributes",options="header"]
|===
|Option
|Default
|Description

|[[embeddings-ollama-url]]xref:embeddings-ollama-url[`embeddings.ollama.url`]
|No default value
|URL of the Ollama server, including port number, for example, `http://localhost:11434`.
|[[embeddings-ollama-model-name]]xref:embeddings-ollama-model-name[`embeddings.ollama.model.name`]
|No default value
|Name of the embedding model.
|[[embeddings-ollama-operation-timeout-ms]]xref:embeddings-ollama-operation-timeout-ms[`embeddings.ollama.operation.timeout.ms`]
|15000 (15 seconds)
|Maximum amount of time in milliseconds to wait for the embeddings reply.
|===

=== ONNX MiniLM

Provides the link:https://docs.langchain4j.dev/integrations/embedding-models/in-process[ONNX in-process] `all-minilm-l6-v2` model, which is included directly in the jar file.
No other configuration besides the general one is needed.

This model is especially suited to prototyping and testing, because it does not depend on any external infrastructure or remote requests. 

=== Voyage AI

Embeddings provided by link:https://www.voyageai.com/[Voyage AI] models.

.Voyage AI embeddings configuration options
[cols="30%a,25%a,45%a",subs="+attributes",options="header"]
|===
|Option
|Default
|Description

|[[embeddings-voyageai-access-token]]xref:embeddings-voyageai-access-token[`embeddings.voyageai.access.token`]
|No default value
|The Voyage AI access token.
|[[embeddings-voyageai-model-name]]xref:embeddings-voyageai-model-name[`embeddings.voyageai.model.name`]
|No default value
|Name of the embedding model.
The list of Voyage AI models can be found in the link:https://docs.voyageai.com/docs/embeddings[Voyage AI Text Embeddings documentation].
|[[embeddings-voyageai-baseurl]]xref:embeddings-voyageai-baseurl[`embeddings.voyageai.baseUrl`]
|https://api.voyageai.com/v1/
|Base Voyage AI API server.
|[[embeddings-voyageai-operation-timeout-ms]]xref:embeddings-voyageai-operation-timeout-ms[`embeddings.voyageai.operation.timeout.ms`]
|15000 (15 seconds)
|Maximum amount of time in milliseconds to wait for the embeddings reply.
|===