8/20/2023 Admin

What Are Embeddings and Vectors (And Why Should I Care?)

(A series of embedding vectors plotted by http://projector.tensorflow.org/)

An embedding is a way to represent a piece of text that captures what it means. It is represented as a vector of floating-point numbers where each element has a specific value and position within the vector. The relationships between these values help to represent the meaning of the original text in a way that machine learning models can understand.

If two embeddings are close together, it means that the texts they represent are similar in meaning. Embeddings can help computers understand how different pieces of information are related to each other.

One common method to measure the similarity between vectors is cosine similarity. This will be demonstrated in the following sample code. By using cosine similarity, you can determine that the vectors for “car” and “tire” are close together, indicating that these words are semantically related. In this way, embeddings provide a powerful tool for computing similarity between documents and queries, enabling effective natural language processing applications.

The RAG Pattern

As described in the article: Use a Poor Developers Vector Database to Implement The Retrieval Augmented Generation (RAG) Pattern, the RAG pattern is a technique for building natural language generation systems that can retrieve and use relevant information from external sources.

The concept is to first retrieve a set of passages that are related to the search query, then use them to supply grounding to the prompt, to finally generate a natural language response that incorporates the retrieved information. Embeddings and vectors are the primary tool used to perform the search needed and to return the relevant chunks of text.

The Sample Application

The sample application allows the end user to enter a series of words or phrases and then click the Create Embeddings button to create the embeddings and vectors.

If we set a break point in the code and examine the values at this point, we see that each word, in this example coffee, has an embedding created that is a series of 1536 float values.

We can expand the collection to see the actual values.

If we let the application run at this point, a page is displayed that allows the end-user to enter a word or a phrase and to click the Search button.

The word or phrase that is entered is also converted to an embedding, and its vectors are calculated against the previous embeddings, using the cosign similarity calculation.

The results of the calculation, that produce a match percentage, is then displayed.

Notice that when the word latte is entered, the words milk and coffee are the closest match, while car is at the end of the matching list.

However, if we enter the phrase Pulling into a parking space (this is turned into a single embedding) the words car and tire are now the closest matches.

Also note that in both cases, the words are semantically matched. Meaning, it is the definition of the words that are matched.

Setting Up Azure OpenAI

See the article: What Is Azure OpenAI And Why Would You Want To Use It? for instructions on gaining access and setting up Azure OpenAI.

Follow the Deploy A Model section in that article for instructions on how to deploy a model.

In this case, select the text-embedding-ada-002 model and give it the Deployment name of text-embedding-ada-002.

This model will be used to obtain the embeddings that will provide the vector data for the embeddings.

Select the model deployment, select Edit deployment, expand Advanced options, and set the Tokens per Minute to at least 120K.

We use this model to create the embeddings, rather than a more advanced model such as GPT 3.5 or GPT 4, because this model has been optimized to create embeddings and is faster and less expensive. Also note that all models create a unique set of embeddings. Therefore, the model used to create embeddings, must be the same model used when embedding other words and phrases to compare to those embeddings.

Create The Application

We will start with the code for the project AzureOpenAIChat created in the article Creating A Blazor Chat Application With Azure OpenAI (you can download the code from the Downloads page on this site).

Open the project in Visual Studio and add a page called Embeddings.razor using the following code:

@page "/embeddings"
@using Azure.AI.OpenAI;
@using Azure;
@inject IConfiguration _configuration
<PageTitle>Embeddings</PageTitle>
@code {
    string Endpoint = "";
    string DeploymentOrModelName = "";
    string Key = "";
    string words_to_embed = 
    "coffee\nmilk\nhot water\nchocolate\ntea\nchai\ncar\ntire";
    string prompt = "latte";
    public class Embedding
    {
        public string? Text { get; set; }
        public float[]? Values { get; set; }
    }
    List<Embedding> embeddingCollection = new List<Embedding>();
    List<(string?, float)> similarities = new List<(string?, float)>();
}

The User Interface

Use the following code for the user interface. Place it before the @code section:

<h1>Embeddings</h1>
@if (embeddingCollection.Count == 0)
{
    <div>
        <textarea rows="10" cols="25"
                  style="vertical-align:text-top"
        @bind="words_to_embed"></textarea>
        <br /><br />
        <button class="btn btn-success"
        @onclick="CreateEmbeddings">
            Create Embeddings
        </button>
    </div>
}
else
{
    <div>
        <textarea rows="2" cols="25"
                  style="vertical-align:text-top"
        @bind="prompt"></textarea>
        <br /><br />
        <button class="btn btn-success"
                style="vertical-align:text-top"
        @onclick="CallService">
            Search
        </button>
    </div>
    <br />
    <ul>
        @foreach (var item in similarities)
        {
            <li>
                @item.Item1 - @item.Item2
            </li>
        }
    </ul>
}

This displays a list box to capture the words or phrases to be embedded, and then displays a text box to capture a word or phrase to be used in a cosign similarity search. It then displays the results of that search.

The Code

Enter the following method in the @code section:

    // Initialize the component by setting the organization and API key
    protected override void OnInitialized()
    {
        // Get the Azure OpenAI Service configuration values
        Endpoint =
        _configuration["AzureOpenAIServiceOptions:Endpoint"] ?? "";
        DeploymentOrModelName =
        _configuration["AzureOpenAIServiceOptions:DeploymentOrModelName"] ?? "";
        Key =
        _configuration["AzureOpenAIServiceOptions:Key"] ?? "";
    }

This retrieves the required Azure OpenAI values from the appsettings.json file or the secrets.json file. See the article Creating A Blazor Chat Application With Azure OpenAI for instructions on configuring this.

Next add the following method that will call the text-embedding-ada-002 model in Azure OpenAI to create the embeddings when the user enters a series of words or phrases and clicks the Create Embeddings button:

   // Create embeddings for the given terms using the OpenAI API
   private void CreateEmbeddings()
   {
       // Get the words_to_embed from the text area
       var colWords = words_to_embed.Split("\n");
       // Create a new OpenAIClient object
       // with the provided API key and Endpoint
       OpenAIClient client = new OpenAIClient(
           new Uri(Endpoint),
           new AzureKeyCredential(Key));
       // loop through each word and create an embedding
       foreach (var word in colWords)
       {
           // Create embeddings for each term
           var embedding =
           client.GetEmbeddings(
               "text-embedding-ada-002",
               new EmbeddingsOptions(word)
           );
           // Add the embedding to the embedding collection
           embeddingCollection.Add(
               new Embedding
                   {
                       Text = word,
                       Values = embedding.Value.Data[0]
                       .Embedding.Select(d => (float)d).ToArray()
                   });
       }
   }

Next, enter the method that is called when the user enters a word or phrase to perform a semantic search:

    // Call the OpenAI service to calculate similarities between embeddings
    private void CallService()
    {
        // Reset the similarities list
        similarities = new List<(string?, float)>();
        // Create a new OpenAIClient object
        // with the provided API key and Endpoint
        OpenAIClient client = new OpenAIClient(
            new Uri(Endpoint),
            new AzureKeyCredential(Key));
        // Create an embedding for the user's input prompt
        var EmbeddingQueryResponse =
        client.GetEmbeddings(
            "text-embedding-ada-002",
            new EmbeddingsOptions(prompt)
        );
        // Calculate the similarity between the prompt's
        // embedding and each existing embedding
        foreach (var embedding in embeddingCollection)
        {
            if (embedding.Values != null)
            {
                var similarity =
                CosineSimilarity(
                    EmbeddingQueryResponse.Value.Data[0].Embedding
                    .Select(d => (float)d).ToArray(),
                    embedding.Values);
                similarities.Add((embedding.Text, similarity));
            }
        }
        // Sort the results by similarity in descending order
        similarities.Sort((a, b) => b.Item2.CompareTo(a.Item2));
    }

This method calls the CosineSimilarity function. Implement that function using the code below:

   private float CosineSimilarity(float[] vector1, float[] vector2)
   {
       // Initialize variables for dot product and
       // magnitudes of the vectors
       float dotProduct = 0;
       float magnitude1 = 0;
       float magnitude2 = 0;
       // Iterate through the vectors and calculate
       // the dot product and magnitudes
       for (int i = 0; i < vector1?.Length; i++)
       {
           // Calculate dot product
           dotProduct += vector1[i] * vector2[i];
           // Calculate squared magnitude of vector1
           magnitude1 += vector1[i] * vector1[i];
           // Calculate squared magnitude of vector2
           magnitude2 += vector2[i] * vector2[i];
       }
       // Take the square root of the squared magnitudes
       // to obtain actual magnitudes
       magnitude1 = (float)Math.Sqrt(magnitude1);
       magnitude2 = (float)Math.Sqrt(magnitude2);
       // Calculate and return cosine similarity by dividing
       // dot product by the product of magnitudes
       return dotProduct / (magnitude1 * magnitude2);
   }