O evento foi conduzido peloDheeraj Bandaru, que é Product Manager do Azure Static Web Apps na Microsoft. Dheeraj contou com a participação doSkyler Hardle, que é Senior Manager do Apps Spaces na Microsoft.
Skyler, apresentou um guia prático sobre a integração entre esses serviços e a implementação de uma aplicação RAG (Retrieval Augmented Generation). A seguir, compartilharei com vocês o que foi abordado durante o evento, detalhando a arquitetura da aplicação e as etapas de desenvolvimento.
Mas, antes precisamos entender esse novo serviço e como ele pode ser útil para a criação de aplicações modernas.
Inclusive, se você deseja assistir ao evento na íntegra, poderá acessar o link abaixo:
O que é o App Spaces?
App Spacesé um serviço inteligente para desenvolvedores que reduz a complexidade de criar e gerenciar aplicações web. Ele ajuda a identificar os serviços corretos para suas aplicações no Azure e proporciona uma experiência de gerenciamento simplificada para o processo de desenvolvimento.
App Spacestambém oferece os benefícios de implantar uma aplicação através dos serviços existentes do Azure, comoAzure Container AppseAzure Static Web Apps, com uma experiência focada no desenvolvimento e implantação de forma simplificada e rápida.
Resumindo….
App Spaces é a maneira mais rápida de implantar e gerenciar suas cargas de trabalho de aplicativos inteligentes, sem ser um especialista em nuvem.
O que é Geraçaõ Aumentada por Recuperação (RAG)?
Geração Aumentada por Recuperação (RAG)é uma técnica que combina a recuperação de informações externas com a geração de respostas por modelos de linguagem (LLMs).
Essa abordagem permite enriquecer o contexto dos modelos de IA, proporcionando respostas mais precisas e relevantes. No caso do nosso aplicativo, utilizamos bancos de dados vetoriais para armazenar documentos e embeddings, permitindo uma recuperação eficiente e contextualizada de informações.
Primeiro, alguns conceitos básicos sobre RAG
Os grandes modelos de linguagem (LLMs) como o ChatGPT são treinados com dados públicos disponíveis na internet no momento do treinamento. Eles podem responder a perguntas relacionadas aos dados nos quais foram treinados. No entanto, esses dados públicos podem não ser suficientes para atender a todas as suas necessidades.
Você pode querer respostas baseadas em seus dados privados ou os dados públicos podem simplesmente estar desatualizados. A solução para esse problema é a Geração Aumentada por Recuperação (RAG), um padrão usado em IA que utiliza um LLM para gerar respostas com base em seus próprios dados.
Como o RAG Funciona?
RAG é um padrão que usa seus dados com um LLM para gerar respostas específicas aos seus dados. Quando um usuário faz uma pergunta, o armazenamento de dados é pesquisado com base na entrada do usuário. A pergunta do usuário é então combinada com os resultados correspondentes e enviada ao LLM usando um prompt (instruções explícitas para um modelo de IA ou aprendizado de máquina) para gerar a resposta desejada. Isso pode ser ilustrado da seguinte forma.
A imagem abaixo representa o fluxo de trabalho do RAG:
No evento, Skyler apresentou a arquitetura do aplicativo RAG que seria desenvolvido durante a apresentação. um frontend em React, um backend em FastAPI e um banco de dados vetorial. Esses componentes trabalham juntos para fornecer uma solução completa e integrada. Abaixo, detalho o funcionamento de cada um desses componentes.
A seguir, detalharei cada um desses componentes.
Frontend com React
O frontend do aplicativo é desenvolvido em React e faz chamadas a um backend FastAPI para buscar e armazenar embeddings em um banco de dados vetorial (Quadrant). A interface do usuário permite inserir perguntas e visualizar as respostas geradas pelo sistema.
Backend com FastAPI
O backend é responsável por duas funcionalidades principais:
Gerar e armazenar embeddings: Utiliza a API de embeddings do OpenAI para gerar vetores a partir do texto fornecido e armazená-los no banco de dados vetorial.
Recuperar e gerar respostas: Busca embeddings relevantes no banco de dados vetorial e utiliza o OpenAI GPT-4 para gerar respostas baseadas nesses embeddings.
Banco de Dados Vetorial
Para a aplicação foi utilizado o Quadrant, um banco de dados vetorial que permite armazenar e recuperar embeddings de forma eficiente. O Quadrant é uma solução escalável e de alto desempenho para aplicações que requerem a recuperação de informações relevantes com base em consultas feitas pelo usuário.
Implementação da Aplicação RAG
Configuração Inicial
Para começar, acesse o App Spaces no portal Azure e selecione a opção para criar um novo ‘space’ de aplicação. Você pode escolher um repositório do GitHub ou utilizar um dos templates fornecidos, como o RAG App. Configure as variáveis de ambiente necessárias, incluindo as chaves da API do OpenAI.
Implantação do Frontend
O frontend da aplicação é uma aplicação React que pode ser implantado no Azure Static Web Apps. Para isso, basta configurar o repositório do GitHub e definir o diretório de saída da aplicação. O Azure Static Web Apps irá criar automaticamente um pipeline de CI/CD para implantar a aplicação.
Configuração e Implantação do Backend
O backend em FastAPI é implantado usando o Azure Container Apps no Azure. Ele manipula as requisições de geração e armazenamento de embeddings, bem como a recuperação e geração de respostas.
Demonstração da Aplicação RAG
Inserindo e Armazenando Texto:O usuário insere um texto no frontend, que é enviado ao backend para gerar e armazenar embeddings no banco de dados vetorial.
Fazendo Perguntas e Recebendo Respostas:O usuário faz perguntas através do frontend, que são processadas pelo backend para buscar embeddings relevantes e gerar respostas utilizando o GPT-4 da OpenAI.
Se desejar saber mais da aplicação desenvolvida no evento, poderá acessar o repositório do GitHubapp-spaces-rag-app
Inclusive,Julia Muiruri, que é Developer Advocate na Microsoft, gravou um vídeo sobre a aplicação desenvolvida no evento. Você pode assistir o vídeo abaixo:
Gerenciamento e Extensibilidade
App Spaces facilita a adição de novos componentes a sua aplicação, como novos frontends ou integrações com outros serviços. A interface de gerenciamento permite visualizar logs, métricas e configurar variáveis de ambiente de forma intuitiva.
O mais interessante disso tudo é que independente da linguagem de programação no lado do Backend, você pode utilizar o App Spaces para gerenciar e implantar sua aplicação. Seja você um desenvolvedor Python, .NET, Node.js, Java, Go, etc., o App Spaces é uma excelente opção para gerenciar suas aplicações.
Se desejar ver outros exemplos de aplicações em diferentes linguagens de programação, poderá acessar o repositório do GitHubapp-spaces-samples.
E, já recomendo em você começar agora a testar o App Spaces! Você pode começar a testar de forma totalmente gratuita. Acesse agora: AQUI ou clicando na imagem abaixo:
Conclusão
No encontro da comunidade Azure Static Web Apps, aprendemos a criar uma aplicação de Geração Aumentada por Recuperação (RAG) utilizando Azure Static Web Apps e App Spaces. Exploramos a arquitetura da aplicação, configuramos os componentes necessários e implementamos uma solução prática que enriquece o contexto dos modelos de IA, proporcionando respostas mais precisas e relevantes.
Lembrando que você poder rever o evento na íntegra, acessando o link abaixo:
Microsoft Reactor YouTube Channel
Você sabia que temos uma das maiores comunidades com temas diversos sobre tecnologia, nuvem e desenvolvimento de software? Não! Então, você precisa conhecer o Microsoft Reactor.
Além disso conta com inúmeros eventos online, presenciais, workshops, meetups e muito mais. Acesse o link da página oficial do Microsoft Reactor e inscreva-se no canal para ficar por dentro de tudo que acontece na comunidade.
Recursos Adicionais
Gostou do App Spaces? Quer saber mais sobre o serviço e como ele pode ajudar a simplificar o desenvolvimento de aplicações web? Ou melhor, quer aprender através de um tutorial prático como criar uma aplicação com App Spaces?
Deixarei abaixo alguns links que poderão te ajudar a explorar mais sobre o App Spaces e como ele pode ser útil para o desenvolvimento de aplicações modernas.
This article is contributed. See the original author and article here.
Microsoft Copilot for Sales is reimagining sales. Integrated seamlessly into your daily tools across Microsoft 365 and Teams, Copilot for Sales harnesses the power of generative AI and customer data to keep sellers in their flow of work so they can spend more time with customers.
Capabilities highlighted below are included in the July release of Copilot for Sales. It may take time for specific capabilities to reach every tenant in each market.
In Outlook
Share key email information easily from Outlook to Teams
Today, sellers face challenges in efficiently sharing crucial customer information received via Outlook with their sales team members on Teams. Currently, they resort to manual copying and pasting of information, searching for the appropriate recipient, and then initiating a chat or posting in the right channel on Teams to share it.
With this release, we’re introducing seamless sharing to Teams in the seller’s flow of work in Outlook to help sales teams quickly start a Teams discussion with up-to-date and relevant information that can move a deal forward. Now, sellers can:
Share key email information with their colleagues in Teams
Add preamble text before sharing
Review and modify key email info before sharing
Share to any location in Teams ─ with a person, in a group chat, or in a Teams channel
Example of user flow for sharing key email info to Teams using Copilot for Sales
Sellers are busy and today updating information crucial for their businesses directly in their CRM requires manual work. Bringing the ability to update their CRM to the flow of work for quick updates, sellers can save time and be more efficient.
This month, we’re excited to announce the rollout of additional language support for suggested CRM updates. Prior to this release, four languages were supported; this expansion enables Copilot for Sales to provide suggested updates in ten languages* for:
Monetary value changes (like budgets, revenue, fees etc.)
The fields a seller can update depend on their CRM permissions as well as which editable fields are enabled by the admins. Calculate fields, and all other scenarios will deep link to their CRM.
Screenshot of suggested CRM updates user interface
In Teams
Stay ahead of activities with meeting prep notes and recent emails
Sellers and sales managers preparing for a meeting sift through vast amounts of data from a variety of sources, including their CRM, a time consuming and sometimes overwhelming activity. This can lead to information overload and potentially cause difficulty in prioritizing the most relevant details for their meeting.
We’re pleased to announce two new capabilities in our meeting prep to help both sellers and sales managers stay ahead of their most impactful activities! Starting with our July release:
A meeting prep card will be sent about an hour* before a meeting instead of just as the meeting starts. (admin configuration settings coming soon!)
Recent emails in meeting prep now contain information about the last 2 emails sharing the same external contacts who are participating in the meeting.
*card will appear 45-75 minutes prior to meeting start time
Screenshot of meeting prep info card inside Teams.
Get started
Ready to join us and other top-performing sales organizations worldwide? Reach out to your Microsoft sales team or visit our product web page.
Ready for the latest tips…and more? Copilot for Sales Tip Time can serve as a foundation for your training of Copilot for Sales users, customers, or partners! This content includes use cases and demonstrates how each feature will benefit sellers, administrators, and sales managers.
Looking for the latest adoption resources? Visit the Copilot for Sales Adoption Center and find the latest information about how to go from inspiration to adoption.
This article is contributed. See the original author and article here.
In today’s digital landscape, ensuring the safety and integrity of AI-generated content is paramount. Azure Content Safety, combined with Azure API Management, provides a robust solution for managing and securing Azure OpenAI endpoints. This blog will guide you through the integration process, focusing on text analysis and prompt shields.
What is Azure Content Safety?
Azure AI Content Safety provides analysis for user and AI generated content. Currently available APIs include
Prompt Shields: scans user text and document text for input attacks on LLMs
Groundedness Detection: Verify if the response generated by the LLMs are grounded in the source provided
Protected material text detection: checks for existence of copyrighted material in the AI generated response
Analyze Text/Image: identifies and categorizes text severity against sexual content, hate, violence and self-harm
Why Integrate Azure Content Safety?
Azure Content Safety offers advanced algorithms to detect and mitigate harmful content in both user prompts and AI-generated outputs. By integrating this with Azure API Management, you can:
Enhance Security: Protect your applications from harmful content.
Ensure Compliance: Adhere to regulatory standards and guidelines.
Improve User Experience: Provide a safer and more reliable service to your users.
Onboard Azure Content Safety API to Azure API Management
Like any other APIs, we can onboard Azure Content Safety API to Azure APIM by importing the latest OpenAPI specification. API management helps with enabling Managed Identity based authentication to the Content Safety API as well as communicate privately using Private Endpoints.
Onboard Azure OpenAI to Azure API Management
Onboarding AOAI to API Management comes with many benefits which are extensively discussed. I have a blog and github repo that talks about this in detail.
Integrate Content Safety with Azure OpenAI APIs in API Management
AI Gateway Labs is an amazing repository exploring various patterns through a series of labs. We have included 2 Content Safety scenarios as labs to demonstrate this integration.
The pattern behind this integration is to leverage the send-request policy in APIM to invoke the respective Content Safety API, and decide to forward the request to OpenAI, if it is safe.
The snippet below concatenates all the prompts in the incoming request to OpenAI and validates if there is any attack detected.
<send-request mode="new" response-variable-name="safetyResponse">
<set-url>@("https://" + context.Request.Headers.GetValueOrDefault("Host") + "/contentsafety/text:shieldPrompt?api-version=2024-02-15-preview")</set-url>
<set-method>POST</set-method>
<set-header name="Ocp-Apim-Subscription-Key" exists-action="override">
<value>@(context.Variables.GetValueOrDefault<string>("SubscriptionKey"))</value>
</set-header>
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-body>@{
string[] documents = new string[] {};
string[] messages = context.Request.Body.As<JObject>(preserveContent: true)["messages"].Select(m => m.Value<string>("content")).ToArray();
JObject obj = new JObject();
JProperty userProperty = new JProperty("userPrompt", string.Concat(messages));
JProperty documentsProperty = new JProperty("documents", new JArray(documents));
obj.Add(userProperty);
obj.Add(documentsProperty);
return obj.ToString();
}</set-body>
</send-request>
<choose>
<when condition="@(((IResponse)context.Variables["safetyResponse"]).StatusCode == 200)">
<choose>
<when condition="@((bool)((IResponse)context.Variables["safetyResponse"]).Body.As<JObject>()["userPromptAnalysis"]["attackDetected"] == true)">
<!-- Return 400 if an attach is detected -->
<return-response>
<set-status code="400" reason="Bad Request" />
<set-body>@{
var errorResponse = new
{
error = new
{
message = "The prompt was identified as an attack by the Azure AI Content Safety service."
}
};
return JsonConvert.SerializeObject(errorResponse);
}</set-body>
</return-response>
</when>
</choose>
</when>
<otherwise>
<return-response>
<set-status code="500" reason="Internal Server Error" />
</return-response>
</otherwise>
</choose>
The snippet below concatenates all the prompts in the incoming request to OpenAI and validates if it is within the allowed limits for hate, sexual, self harm and violence.
<send-request mode="new" response-variable-name="safetyResponse">
<set-url>@("https://" + context.Request.Headers.GetValueOrDefault("Host") + "/contentsafety/text:analyze?api-version=2023-10-01")</set-url>
<set-method>POST</set-method>
<set-header name="Ocp-Apim-Subscription-Key" exists-action="override">
<value>@(context.Variables.GetValueOrDefault<string>("SubscriptionKey"))</value>
</set-header>
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-body>@{
string[] categories = new string[] {"Hate","Sexual","SelfHarm","Violence"};
JObject obj = new JObject();
JProperty textProperty = new JProperty("text", string.Concat(context.Request.Body.As<JObject>(preserveContent: true)["messages"].Select(m => m.Value<string>("content")).ToArray()));
JProperty categoriesProperty = new JProperty("categories", new JArray(categories));
JProperty outputTypeProperty = new JProperty("outputType", "EightSeverityLevels");
obj.Add(textProperty);
obj.Add(categoriesProperty);
obj.Add(outputTypeProperty);
return obj.ToString();
}</set-body>
</send-request>
<choose>
<when condition="@(((IResponse)context.Variables["safetyResponse"]).StatusCode == 200)">
<set-variable name="thresholdExceededCategory" value="@{
var thresholdExceededCategory = "";
// Define the allowed threshold for each category
Dictionary<string, int> categoryThresholds = new Dictionary<string, int>()
{
{ "Hate", 0 },
{ "Sexual", 0 },
{ "SelfHarm", 0 },
{ "Violence", 0 }
};
foreach (var category in categoryThresholds)
{
var categoryAnalysis = ((JArray)((IResponse)context.Variables["safetyResponse"]).Body.As<JObject>(preserveContent: true)["categoriesAnalysis"]).FirstOrDefault(c => (string)c["category"] == category.Key);
if (categoryAnalysis != null && (int)categoryAnalysis["severity"] > category.Value)
{
// Threshold exceeded for the category
thresholdExceededCategory = category.Key;
break;
}
}
return thresholdExceededCategory;
}" />
<choose>
<when condition="@(context.Variables["thresholdExceededCategory"] != "")">
<return-response>
<set-status code="400" reason="Bad Request" />
<set-body>@{
var errorResponse = new
{
error = new
{
message = "The content was filtered by the Azure AI Content Safety service for the category: " + (string)context.Variables["thresholdExceededCategory"]
}
};
return JsonConvert.SerializeObject(errorResponse);
}</set-body>
</return-response>
</when>
</choose>
</when>
<otherwise>
<return-response>
<set-status code="500" reason="Internal Server Error" />
</return-response>
</otherwise>
</choose>
Conclusion
Integrating Azure Content Safety with API Management for Azure OpenAI endpoints is a powerful way to enhance the security and reliability of your AI applications. By following these steps, you can ensure that your AI-generated content is safe, compliant, and user-friendly.
This article is contributed. See the original author and article here.
Searching for content on a website uses keyword-based search, a method with its limitations. For example, searching for a ‘rain jacket for women’ on an e-commerce website returns all jackets for both men and women, as the search focuses on just the keyword ‘jacket’. The search simply isn’t contextual enough to provide the desired results.
Semantic search is a technique that allows users to search for information using natural language queries rather than specific keywords. With semantic search, the meaning and intent of users’ queries are inferred, and personalized relevant results are returned. Generative AI, on the other hand, is a type of artificial intelligence that can generate new content from existing data, such as text, images, or audio. You can use generative AI to produce summaries, captions, recommendations, or responses based on a user’s input and preferences.
This blog post discusses how to build intelligent MySQL applications with semantic search and generative AI response using Azure Open AI and Azure Database for MySQL with Azure AI search. As an example, we’ll use a Magento ecommerce app designed to sell jackets, and then build a “Product Recommender CoPilot” chat application that:
Recognizes the intent of a user’s natural language queries.
Generates custom responses to recommend suitable products using the product details and reviews data stored in Azure Database for MySQL.
Architecture
The simplest way to include the rich capabilities of semantic search and generative AI in your applications is to build a solution using the Retrieval Augmented Generation (RAG) architecture with Azure AI Search and Azure Open AI services.
What is a RAG architecture?
Retrieval Augmented Generation, or RAG, is an architecture that augments the natural language understanding and generation capabilities of LLMs like ChatGPT by adding an information retrieval system like Azure AI Search which works with your data stored in data sources like Azure Database for MySQL. In a typical RAG pattern:
A user submits a query or prompt in natural language.
The query is routed to Azure AI Search to find the relevant information.
Azure AI Search sends the top ranked semantic search results to a Large Language Model (LLM).
The LLM then processes the natural language query and uses reasoning capabilities to generate a response to the initial prompt.
Sample product recommender Copilot architecture
A sample RAG architecture for the AI solution we’ll show you how to build in this blog post appears in the following graphic:
Azure AI search pulls the content (in our case, the product details and reviews data) from a backend Azure Database for MySQL database by using an indexer that runs periodically.
The product details and reviews data are further chunked and vectorized using Azure OpenAI’s text embedding model.
Azure AI Search then persists this vectorized data in a vector search index.
When a user uses the “Product Recommender CoPilot” chat application, the query is sent to an Azure OpenAI Chat Completion Service.
Azure AI Search is now used as a data source to find the most relevant response using vector-search or hybrid search (vector + semantic search).
The Azure OpenAI Chat Completion service then uses these search results to generate a custom response back to the user query.
In this post, we’ll walk you through how to set up the backend data sources, indexers, and models required to build this solution. It is a detailed guide to the sample Python code hosted in our GitHub repository in a Jupyter Notebook: azure-mysql/Azure_MySQL_AI_Search_Sample at master Azure/azure-mysql.
For a short demo of the entire process using a Magento ecommerce application, watch the following video!
Prerequisites
Before getting started, you need to ensure that the following prerequisites are in place.
An Azure account with an active subscription. If you don’t have one, create one for free here.
An Azure AI Search resource. If you don’t have one, you can create one via the Azure portal or the Azure CLI, as explained in the article here.
An Azure Open AI Services resource. If you don’t have one, you can create one via the Azure portal or the Azure CLI, as explained in the article here.
A MySQL database (in Azure Database for MySQL or any database provider) populated with product and reviews data obtained from your ecommerce application like Magento.
To create an Azure Database for MySQL server, follow the instructions in the article here.
Supporting semantic search in an e-commerce application that leverages Azure AI search and Azure Open AI Services in the backend requires completing the following:
I. Set up data source connection in Azure AI Search.
II. Set up automatic chunking, vectorization and indexing.
III. Use vector search from a sample application.
IV. Generate a GPT response to the user.
V. Test the solution in the Azure OpenAI Studio playground.
Azure AI search pulls the contents and reviews data from a backend MySQL flexible server by using an indexer that runs periodically. The reviewed data is further chunked and vectorized using Azure OpenAI’s text embedding model. In Azure AI search, the vectorized data then persists in a vector search index.
I. Set up data source connection in Azure AI Search
The data source definition specifies the data to index, credentials, and policies for identifying changes in the data. The data source is defined as an independent resource so that it can be used by multiple indexers. In this example, we’ll use a custom table with product details and review data which is stored in a database in Azure Database for MySQL.
In the Azure AI service, before creating a search Index, we’ll need to create a connection to your data source. We’ll import Azure AI classes like ‘SearchClient’, ‘SearchIndexerClient’, ‘SearchIndexerDataSourceConnection’ and their functions like “create_or_update_data_source_connection()” to setup the data source connection in Azure AI Search. We’ll also import several other models – the comprehensive list is shared in the following code sample.
II. Set up automatic chunking, vectorization and indexing
We are now ready to create an Azure Database for MySQL indexer that periodically pulls the product details and reviews data from the database in Azure Database for MySQL, chunks and vectorizes the data and persists it in a vector search index.
To do this, we’ll first create an index which takes the product details and reviews data for each product, splits the combined text into chunks and embeds each chunk as a vector. For any incoming user query, we’ll search for the most relevant chunk using vector search and semantic search.
Create an Azure AI Search index
In Azure AI Search, a search index available to the search engine for indexing, full text search, vector search, hybrid search, and filtered queries. An index is defined by a schema and is saved to the search service.
We’ll first create an index object from the ‘SearchIndexClient’ class.
Then, we define the field mappings to correlate the fields in a MySQL database to the fields in the AI search Index. As the combined text is generally long, it needs to be chunked into smaller words. To do this, we’ll add an additional search field called “chunk”.
Next, we’ll decide the searches that the index will support. In this example, we’ll use ‘vector search’ along with ‘semantic re-ranking’. Here is how the two work together in our solution:
Vector search is first performed on all the entries in the search index.
Semantic search is sort of a neural network where the search can be performed only of limited number. The top 50 results obtained using vector search is sent to the neural network, which re-ranks these documents and provides the top matched result in the reduced context. Semantic search does a better optimization to find the best results. It also produces short-form captions and answers that are useful as LLM inputs for generation.
We’ll then define the vector search and semantic configurations:
Vector search configuration – You can choose different algorithms, such as HNSW or KNN, to perform the vector search. In this post, we’ll choose the most used algorithm – HNSW. We’ll configure the HNSW algorithm to use the ‘COSINE’ metric. Considering each vector as a point in the multi-dimensional space, the algorithm will not find the cosine distance between the points. Lower the distance, more similar the vectors.
Semantic configuration – Here, we’ll define the index field on which semantic re-ranking is performed.
Finally, we’ll create the search index using the above two configurations on the relevant MySQL DB table fields.
We’ll create a skillset using two pre-built “skills”:
The “Split Skill”, which takes the concatenated text and divides it into chunks.
The “Azure OpenAI Embedding Skill”, which takes the outputs of the Split Skill and vectorizes them individually.
We’ll then apply an Index Projector to make it so that our final index has one item for every chunk of text, rather than one item for every original row in the database.
After you define the data source and create the index, you’re ready to create the indexer. The configuration of the Indexer requires the inputs, parameters, and properties that control run time behaviors.
To create an indexer, we’ll provide the name of the data source, index and skillset that we created in the previous steps. We’ll then run the indexer at periodic intervals.
After the indexer is created, you can use the Azure portal to view the indexer under the Indexers blade in Azure AI Search. You can also run the indexer from the Azure portal or using Azure CLI. The runs can be configured to be ad hoc or scheduled:
Ad hoc – An indexer can be configured to run only once or on-demand. You can use APIs (like we used in the code sample), Azure portal or CLI to achieve this.
Scheduled – You can schedule and indexer to run at a certain time. Go to “Settings” section in the Azure portal view of the indexer, and choose hourly, one time, daily, or any other custom settings.
III. Use vector search from a sample application
With the Azure AI Search indexer ready and running, you can now use the vector search and semantic search capabilities from your application. To call the search function, provide the user query text, required query parameters like the number of nearest neighbors to return as top hits, and the columns or fields in the index to be considered. Also select query type “Semantic” to include both vector search and semantic search and provide the name of semantic search configuration object that you created in section II.
For a user query like “suggest me some rain jackets for women“, this single API call performs vector search and semantic re-ranking, and returns the top ranked results as shown in the following screenshot. You’ll also see the vector search score, semantic re-ranker score, and all the details of the recommended product.
IV. Generate a GPT response to the user
To generate custom responses to the users, we need to simply make a call to the Azure OpenAI chat completion service. To trigger the completion, we first input some text as a “prompt”. The LLM in the service then generates the completion and attempts to match our context or pattern. An example prompt for this scenario could be: “You are an AI assistant that recommends products to people based on the product reviews data matching their query. Your answer should summarize the review text, include the product ID, include the parent id as review id, and mention the overall sentiment of the review.”
To generate the GPT response, we’ll use the Azure OpenAI’s chat.completions.create() API call and supply it with the user query, the Open AI model to be used, the Azure AI Search index as data source, and the prompt.
V. Test the solution in the Azure OpenAI Studio playground
Finally, it’s important to test the solution in Azure Open AI Studio, a user-friendly platform that allows developers to explore cutting-edge APIs and models and to build, test, deploy, and manage AI solutions for use in websites, applications, and other production environments.
For an end-to-end test of the AI solution in the Azure OpenAI Studio playground, perform the following steps:
Go to Azure OpenAI studio, select your subscription and the Azure AI Search resource, and navigate to the “Chat playground”.
In the section, in the Add your data tab, select Add a data source.
In the pop-up window, select Azure AI Search as the data source.
Fill in subscription details, and then choose the Azure AI Search service and Azure AI Search index created in the previous sections.
Enable the Add vector search to this search resource option.
Select the Azure OpenAI – text-embedding-ada-002 model and select Next.
Select Hybrid + semantic search type.
Review all the details and add the data source.
In the Prompt tab under the Setup section, type your prompt message in the System message text box. You can use the sample prompt from the previous step or modify the prompt to add more details, such as product images, in the response.
To test the solution, type a sample query in the chat box and watch your copilot generate a smart recommendation in response!
Now, you’re all set to deploy this Product Recommender CoPilot AI solution to production!
Conclusion
If you’re running applications, such as content management systems (CMS), e-commerce applications, or gaming sites, with data hosted in Azure Database for MySQL, you can enhance your user experience by building generative AI search and chat applications using LLMs available in Azure OpenAI and vector storage and indexing provided by Azure AI Search. Unleash the power of your data hosted on MySQL with the simple and seamless AI integrations on Azure!
If you have any queries or suggestions for more AI-related content, please let us know by contacting us at AskAzureDBforMySQL@service.microsoft.com. We’re also open to collaborating with you on technical collateral! Check out our Contributors initiative at aka.ms/mysql-contributors to learn more.
This article is contributed. See the original author and article here.
In FY25, we have made several changes to Microsoft Funded Engagements, formerly known as Partner Activities, to support deal acceleration and growth. These pre-defined engagements will help you align your go-to-market efforts with Microsoft and build expertise in enabling core use case scenarios. Read on to learn how to access these program investments NOW!
Bookmark the Business Applications Funded Engagements page to stay updated with the latest resources, updates, and incentive announcements:https://aka.ms/BAFundedEngagements
This article is contributed. See the original author and article here.
We’re pleased to announce significant updates to the Sign-ins workbook in the Microsoft Entra admin center, a crucial tool for organizations transitioning from Azure Active Directory Authentication Libraries (ADAL) to Microsoft Authentication Libraries (MSAL). These updates aim to streamline the ADAL migration process by providing comprehensive insights into your ADAL-application-related data.
Why is this Important?
We announced ADAL end of life in June 2020 and stopped supporting any security updates as of June 2023—which means applications using ADAL can’t utilize the latest security features, leaving them vulnerable to future security threats. We strongly recommend migrating any application using ADAL to MSAL to improve the security posture and resilience of authentication and authorization of your client applications.
MSAL supports the latest security features for Microsoft Entra ID like managed identity, Continuous Access Evaluation (CAE), passkeys, and many more. The updated Sign-ins workbook is an essential tool in this transition, providing the necessary insights and data to make informed decisions to execute migration.
What’s new in the Sign-ins workbook?
The Sign-ins workbook is redesigned for admins needing a centralized and more detailed view of applications using ADAL within their tenant. These additional insights can help them identify, investigate, and validate the ADAL applications to successfully migrate to MSAL.
Here’s what you can expect with the latest enhancements:
Comprehensive sign-in log aggregation: The workbook now consolidates logs from various types of sign-in events, including interactive, non-interactive, and service principal sign-ins.
Enhanced data visualization: We updated the report with new aggregated metrics to enable an all-up view of sign-ins across ADAL applications. To aid in your specific analytical needs, the workbook supports the application of custom filters and queries. This flexibility enables you to focus on the information that matters most to your ADAL migration efforts.
Integration with Microsoft Entra recommendations: You can now directly access this Sign-Ins workbook from the ADAL to MSAL recommendation page to dive deep into the list of ADAL applications listed on the recommendation details page. To use the workbooks for Microsoft Entra ID, you need a Microsoft Entra ID tenant with a P1 license.
Prevent identity attacks, ensure least privilege access, unify access controls, and improve the experience for users with comprehensive identity and network access solutions across on-premises and clouds.
Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!
Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!
Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!
Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!
The 2024 Windows Server Summit was held in March and brought three days of demos, technical sessions, and Q&A, led by Microsoft engineers, guest experts from Intel®, and our MVP community. For more videos from this year’s Windows Server Summit, please find the full session list here.
This article focuses on improvements to Active Directory in Windows Server 2025.
What’s new in Active Directory for Windows Server 2025
The AD product group presents and demonstrate some of the new AD capabilities coming in Windows Server 2025, including new functional levels, security enhancements, and improved scalability.
Protecting Active Directory from management plane attacks
Mind the management plane! Whether your organization is running Active Directory on-premises, hybrid, or fully in the cloud, virtualized domain controllers are almost always present. But when is the last time you checked to ensure your privileged access model, aka Tier 0, extended to encompass the management plane?
Explore the common modern deployment scenarios for virtualized domain controllers and examine the relationship with the management plane. Why? Because attackers can exploit a weakly implemented privileged model and use the management plane as an easy back door into Active Directory.
In this session, we explore scenarios where organizations can unknowingly leave the door open to these attacks, diving deep into commonly observed gaps, and walking through a demonstration of using the management plane as a means of pivoting into Active Directory. Learn how to defend yourself and get actionable recommendations your organization can take today to ensure that the management plane does not become an attacker’s new friend.
The evolution of Windows authentication
As the security landscape evolves, Windows must continue to change to protect users and organizations. Foundational to this is user authentication. In Windows Server 2025 and Windows vNext, we have created completely new Kerberos features to minimize use of NTLM in your environments. This session explains and demonstrates IAKerb, Local KDC, IP SPN, and the roadmap to the end of NTLM.
This article is contributed. See the original author and article here.
We continue to expand the Azure Marketplace ecosystem. For this volume, 182 new offers successfully met the onboarding criteria and went live. See details of the new offers below:
Get it now in our marketplace
AI Assistant: AI Assistant from empower GmbH integrates with Microsoft PowerPoint, Word, and Outlook, providing prebuilt prompts to simplify workflows, improve documents, and define common tasks. You also can create custom prompts for your company’s workflows. The user-friendly interface enables even those without prior AI experience to quickly get started.
AlmaLinux 8.10 on ARM64: This offer from Ntegral provides AlmaLinux 8.10 on a Microsoft Azure virtual machine. Ntegral packages applications following industry standards and continuously monitors all components and libraries for vulnerabilities and application updates.
AspenTech Offering: AspenTech offers solutions for capital-intensive process industries that are facing competitive pressures and sustainability trends. The solutions address challenges in performance engineering, manufacturing, subsurface science, and digital grid management. AspenTech has helped companies optimize assets and reduce energy use.
BlackLine Modern Accounting: BlackLine offers automation solutions for account reconciliation, task management, transaction matching, variance analysis, and more. The platform centralizes data, enhances visibility, and strengthens controls to increase accuracy and efficiency. It also provides real-time dashboards to proactively identify delays and reduce risk.
Cascade: Cascade AI uses large language models to provide accurate and continual human resources support to employees. Using Cascade AI, companies can streamline engagement efforts and realize cost savings. Cascade AI is scalable and can provide multi-language support for all enterprise-wide HR applications.
Commander Edge: Armada provides a full-stack solution for the edge by combining connectivity, compute, storage, and AI/ML applications designed for the edge. The full-stack solution also includes a rugged mobile data center called Galleon.
Compliance Audit Automation: Seconize automates compliance checks for over 50 global regulations, including ISO 27001 and SOC2. Real-time alerts and comprehensive reports ensure adherence to internal and industry standards, while role-based access control allows for efficient auditing.
Conversation: AiWave Conversations from Almawave uses conversational AI and generative AI to improve business interactions through data management, dialogue flow, and the creation of AI assistants. AiWave Conversations manages incoming and outgoing communication channels, transforming the way organizations manage communications and internal processes.
Cribl License (Annual 150 GB): Cribl is a real-time data management and analysis platform that offers data routing, filtering, and transformation tools, as well as monitoring and troubleshooting capabilities. It optimizes data pipelines, improves data quality, and reduces costs associated with data processing.
Crisp Retail Analytics: Real-time: Crisp is a retail data collaboration platform that aggregates, normalizes, and analyzes data from more than 30 retailer and distributor partners. It offers an easy-to-use interactive dashboard that provides daily insights for consumer goods teams across sales, supply chain, and category management.
Debian 10 with Minecraft Java Game Server: This offer from Virtual Pulse S. R. O. provides Debian 10 with Minecraft: Java Edition on a Microsoft Azure virtual machine. Minecraft: Java Edition offers endless creativity and adventure, with diverse biomes, a seamless multiplayer experience, and exceptional performance.
Debian 11 with Minecraft Java Game Server: This offer from Virtual Pulse S. R. O. provides Debian 11 with Minecraft: Java Edition on a Microsoft Azure virtual machine. This is a hassle-free way to start playing Minecraft with preconfigured servers. Players can explore various mechanics and features, build their dream house, mine resources, and engage in multiplayer activities.
Debian 12 Minimal with iPerf3 Server: This offer from Virtual Pulse S. R. O. provides an iPerf3 server along with a minimal installation of Debian 12 on a Microsoft Azure virtual machine. iPerf3 is a powerful tool for network analysis, offering flexibility and control over measurements. Its consolidated error handling simplifies identification and correction of network problems. It’s ideal for testing data migration programs and checking throughput of cloud systems.
Debian 12 with iPerf3 Server: This offer from Virtual Pulse S. R. O. provides an iPerf3 server along with Debian 12 on a Microsoft Azure virtual machine. iPerf3 is a powerful tool for network analysis, offering flexibility and control over measurements. Its consolidated error handling simplifies identification and correction of network problems. It’s ideal for testing data migration programs and checking throughput of cloud systems.
Digital Human Video Maker: This platform as a service from Sensecore Technology Limited generates digital avatars for AI video creation and live-streaming. Input content as text, then select a desired virtual avatar. The avatar will read your text, outputting a video file or a video stream.
Edge AI: Armada’s Edge AI suite offers AI apps for real-time decision-making at the edge, supporting industries such as health and safety, production optimization, and logistics. It caters to the commercial sector and the public sector.
Embee Customer and Identity Access Management: Embee provides secure multifactor authentication and conditional access policies to protect access to applications and resources across datacenters and the cloud. Safeguard your business against potential threats by identifying vulnerabilities in your IT infrastructure.
Engagely AI Total CX Platform: Engagely is a no-code customer experience automation platform for enterprises, powered by generative AI and Microsoft Azure OpenAI Service. It addresses contact center challenges with a strategic approach, offering workflow automation, increased agent productivity, and lower operational costs. Engagely supports more than 120 global languages across various dialects and accents.
Extra360: Extra360 enhances customer engagement and loyalty through promotions, integrated digital wallet solutions, omnichannel engagement, and gamification techniques. The platform has been trusted by hundreds of brands and companies across industries, with demonstrable impact and effectiveness in the marketplace.
F5 NGINX Ingress Controller with F5 NGINX App Protect WAF: NGINX Ingress Controller, based on NGINX Plus, is a high-performance, scalable, and secure modern app solution. NGINX App Protect WAF provides seamless security integration into DevOps environments, with over 6,000 threat signatures and Data Guard for PII masking. It enables scalable application security and governance for web apps, microservices, cloud-native apps, and APIs.
FlatModel: FlatModel lets you easily ingest, transform, and integrate flat file data from various sources into your Microsoft Azure environments. Streamline your operations with automated workflows and scheduling for efficient data processing.
Infosys TradeEdge – Order Orchestration: TradeEdge Order Orchestration enables network-wide inventory access; intelligently splits and routes orders to one or more fulfillment partners based on inventory availability and proximity to customers or consumers; and automates and orchestrates end-to-end fulfillment processes across partners, sales channels, and internal systems.
iPerf3 Server on Debian 11: This offer from Tidal Media provides an iPerf3 server along with Debian 11 on a Microsoft Azure virtual machine. iPerf3 is a powerful tool for analyzing and optimizing network resources. It provides information on maximum throughput, bitrate, loss, and other parameters, helping administrators make informed decisions to improve network performance and reliability.
iPerf3 Server on Oracle 8: i This offer from Tidal Media provides an iPerf3 server along with Oracle Linux 8 on a Microsoft Azure virtual machine. iPerf3 is a powerful tool for analyzing and optimizing network resources. It provides information on maximum throughput, bitrate, loss, and other parameters, helping administrators make informed decisions to improve network performance and reliability.
iPerf3 Server on Red Hat 7: This offer from Tidal Media provides an iPerf3 server along with Red Hat Enterprise Linux 7 on a Microsoft Azure virtual machine. iPerf3 is a powerful tool for analyzing and optimizing network resources. It provides information on maximum throughput, bitrate, loss, and other parameters, helping administrators make informed decisions to improve network performance and reliability.
iPerf3 Server on Red Hat 8: This offer from Tidal Media provides an iPerf3 server along with Red Hat Enterprise Linux 8 on a Microsoft Azure virtual machine. iPerf3 is a powerful tool for analyzing and optimizing network resources. It provides information on maximum throughput, bitrate, loss, and other parameters, helping administrators make informed decisions to improve network performance and reliability.
iPerf3 Server on Ubuntu 20.04: This offer from Tidal Media provides an iPerf3 server along with Ubuntu 20.04 on a Microsoft Azure virtual machine. iPerf3 is a powerful tool for analyzing and optimizing network resources. It provides information on maximum throughput, bitrate, loss, and other parameters, helping administrators make informed decisions to improve network performance and reliability.
iPerf3 Server on Ubuntu 22.04: This offer from Tidal Media provides an iPerf3 server along with Ubuntu 22.04 on a Microsoft Azure virtual machine. iPerf3 is a powerful tool for analyzing and optimizing network resources. It provides information on maximum throughput, bitrate, loss, and other parameters, helping administrators make informed decisions to improve network performance and reliability.
Minecraft Java Game Server on AlmaLinux 8: This offer from Tidal Media provides AlmaLinux 8 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. Take advantage of advanced administrative tools and customization options so players can suit their unique preferences and style.
Minecraft Java Game Server on AlmaLinux 9: This offer from Tidal Media provides AlmaLinux 9 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. This seamless multiplayer environment fosters collaboration and lets players customize their experience with a wide range of options and settings.
Minecraft Java Game Server on Debian 11: This offer from Tidal Media provides Debian 11 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. It’s ideal for communities, gaming enthusiasts, and educational institutions seeking a reliable platform to host multiplayer Minecraft sessions.
Minecraft Java Game Server on Debian 12: This offer from Tidal Media provides Debian 12 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. It offers optimized performance and stability for a smooth multiplayer experience. Comprehensive administrative tools are included.
Minecraft Java Game Server on Oracle 8: This offer from Tidal Media provides Oracle Linux 8 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. Transform textures and lighting with resource packs and shaders to create unique and memorable worlds.
Minecraft Java Game Server on Red Hat 8: This offer from Tidal Media provides Red Hat Enterprise Linux 8 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. Enjoy a stable and secure gaming environment for players of all ages while benefiting from robust infrastructure and security features for server administrators.
Minecraft Java Game Server on Rocky 8: This offer from Tidal Media provides Rocky Linux 8 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. It’s ideal for novice and experienced server administrators alike, and it provides the tools and performance needed to create and maintain a thriving Minecraft community.
Minecraft Java Game Server on Rocky 9: This offer from Tidal Media provides Rocky Linux 9 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. Flexible settings are available for customized gaming experiences, in addition to automatic updates and robust security features.
Minecraft Java Game Server on SUSE 15 SP5: This offer from Tidal Media provides SUSE Linux Enterprise Server 15 SP5 along with Minecraft: Java Edition on a Microsoft Azure virtual machine. Minecraft players, server administrators, and gaming communities can enjoy a reliable and high-performance environment with advanced customization opportunities.
Minecraft Java Game Server on Ubuntu 20.04: This offer from Tidal Media provides Ubuntu 20.04 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. Benefit from automatic updates, robust security features, and scalability to accommodate fluctuating player demand.
Minecraft Java Game Server on Ubuntu 22.04: This offer from Tidal Media provides Ubuntu 22.04 along with a Minecraft: Java Edition server on a Microsoft Azure virtual machine. Use this stable customizable platform for hosting multiplayer Minecraft games, where players can explore procedurally generated worlds, build together in shared virtual environments, and customize server settings and game rules.
MishiPay Scan & Go: MishiPay’s Scan & Go is a mobile self-checkout technology that allows customers to scan, pay, and go, eliminating the need for traditional checkout lines. The app offers real-time information about products, personalized offers, and recommendations. Retailers benefit from increased efficiency, whlie shoppers benefit from convenience, control, and personalization.
Online Course in Productivity with Copilot: Productivity with Copilot is an online course by Founderz Business School that teaches AI in business. The program is for all skill levels and involves microlessons, hands-on assignments, and collaboration. Copilot integrates AI in Windows 11 and Microsoft 365 apps, boosting productivity.
Oracle 8.7 with Minecraft Java Game Server: This offer from Virtual Pulse S. R. O. provides Oracle Linux 8.7 along with a Minecraft: Java Edition game server on a Microsoft Azure virtual machine. Minecraft offers an immersive multiplayer experience in a boundless universe. Explore diverse biomes, build magnificent structures, and engage in thrilling battles.
Oracle Linux 8 Server: This offer from Madarson IT provides an Oracle Linux 8 image preconfigured with GUI/RDP access. Oracle Linux is an enterprise-class Linux distribution designed to provide a stable, high-performance platform for running enterprise applications and workloads. It includes tools and features designed for cloud computing, virtualization, and big data processing, making it suitable for modern datacenters and hybrid cloud environments.
Outcomes & Evaluation for Teams Town Hall: This service from Hive Streaming will provide comprehensive trend analysis for your Microsoft Teams town-hall events over six months. It will evaluate audience engagement, infrastructure performance, and user experience to optimize live communication strategies and help you make informed investment decisions. This offer is intended for IT, network, event, and communications managers.
Rancher Government Multi-Cluster Manager: Rancher’s Multi-Cluster Manager is an open-source platform for Kubernetes that supports Windows containers, Prometheus, and Istio service mesh. It enhances security for cloud-native workloads and is appropriate for organizations implementing Kubernetes in datacenter, cloud, or edge environments.
RBVM Aggregation: The Seconize cybersecurity platform streamlines the auditing process through automation and uses risk-based vulnerability management to allocate resources. Vulnerabilities are categorized as high risk, medium risk, and low risk, and Seconize first focuses on the ones that pose the greatest risk to critical systems or sensitive data.
Red Hat 8.6 with Minecraft Java Game Server: This offer from Virtual Pulse S. R. O. provides Red Hat Enterprise Linux 8.6 along with a Minecraft: Java Edition game server on a Microsoft Azure virtual machine. With dependable performance and a robust moderator community, Minecraft players can enjoy boundless creativity and adventures.
Red Hat 9 with Docker Compose: This offer from Virtual Pulse S. R. O. provides Red Hat Enterprise Linux 9 along with Docker Compose on a Microsoft Azure virtual machine Docker Compose simplifies multi-container app development and deployment. It defines containers in a single YAML file, creating a network for secure internal communication and adapting apps to different environments.
Red Hat Enterprise Linux (RHEL) 8 Minimal with Support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 8 image on a Microsoft Azure virtual machine. The image is stripped down to include only essential packages, making it fast to boot and highly responsive. Use it for web applications, server infrastructure, and development and testing environments.
Red Hat Enterprise Linux (RHEL) 8.10 EUS Minimal with Support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 8.10 image on a Microsoft Azure virtual machine. The image is stripped down to include only essential packages, making it fast to boot and highly responsive. Use it for web applications, server infrastructure, and development and testing environments.
Red Hat Enterprise Linux (RHEL) 8.9 Minimal Gen 2 with Support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 8.9 image on a Generation 2 virtual machine on Microsoft Azure. The image is stripped down to include only essential packages, making it fast to boot and highly responsive. Use it for web applications, server infrastructure, and development and testing environments.
Red Hat Enterprise Linux (RHEL) 9 Gen 2 Minimal with Support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 9 image on a Generation 2 virtual machine on Microsoft Azure. The image is stripped down to include only essential packages, making it fast to boot and highly responsive. Use it for web applications, server infrastructure, and development and testing environments.
Red Hat Enterprise Linux (RHEL) 9.3 Minimal Generation 2 with support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 9.3 image on a Generation 2 virtual machine on Microsoft Azure. The image is stripped down to include only essential packages, making it fast to boot and highly responsive. The image comes with the latest security patches, and Red Hat Update Infrastructure is configured and ready to use without a subscription.
Red Hat Enterprise Linux (RHEL) 9.4 Gen 2 Minimal with Support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 9.4 image on a Generation 2 virtual machine on Microsoft Azure. The image is stripped down to include only essential packages, making it fast to boot and highly responsive. The image comes with the latest security patches, and Red Hat Update Infrastructure is configured and ready to use without a subscription.
Red Hat Enterprise Linux 8 CIS Level 1: This offer from Madarson IT provides a Red Hat Enterprise Linux 8 image hardened to meet Center for Internet Security (CIS) Level 1 compliance needs. CIS controls are internationally recognized cybersecurity best practices that help organizations protect their systems and data against common threats.
Red Hat Enterprise Linux 8 CIS Level 2: This offer from Madarson IT provides a Red Hat Enterprise Linux 8 image hardened to meet Center for Internet Security (CIS) Level 2 compliance needs. Implementing CIS Level 2 benchmarks can help organizations improve their security posture and protect against advanced and persistent threats.
Red Hat Enterprise Linux 9 CIS Level 1: This offer from Madarson IT provides a Red Hat Enterprise Linux 9 image hardened to meet Center for Internet Security (CIS) Level 1 compliance needs. Madarson IT images are always up to date and secure. They follow industry standards and are built to work right out of the box.
Red Hat Enterprise Linux 9 CIS Level 2: This offer from Madarson IT provides a Red Hat Enterprise Linux 9 image hardened to meet Center for Internet Security (CIS) Level 2 compliance needs. Madarson IT images are always up to date and secure. They follow industry standards and are built to work right out of the box.
Red Hat 8.10 EUS Minimal with Support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 8.10 image, stripped down to include only essential packages for optimal performance. The image is updated with the latest security patches and updates, and Red Hat Update Infrastructure is configured and ready to use without a subscription.
Red Hat 9.4 Gen 2 Minimal with Support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 9.4 image on a Generation 2 virtual machine. Gen 2 VMs on Azure use a UEFI-based boot architecture instead of BIOS-based architecture and have improved boot and installation times. It’s ideal for rapid deployment of web applications, efficient development and testing environments, and stable server infrastructure.
Restaurant QR Order system: Tunnelbiz Web Restaurant QR Order System is a cloud-based app that generates QR code menus. Designed specifically for the hospitality industry, it allows multiple restaurants to streamline the ordering process, providing a seamless digital dining experience for patrons and restaurant staff.
RHEL 8.10 EUS Minimal with Support by Rinne Labs: This offer from Rinne Labs provides a lightweight and secure Red Hat Enterprise Linux 8.10 image built from the official ISO image. For optimal performance, it’s stripped down to include only the essential packages. This results in a minimal image that is fast to boot, more responsive, and has lower storage requirements.
RHEL 9.4: This offer from AskforCloud provides Red Hat Enterprise Linux (RHEL) 9.4 on a Microsoft Azure virtual machine. RHEL provides a flexible and stable foundation to support hybrid cloud innovation and a fast, consistent experience for deploying applications and critical workloads.
RHEL 9.4 Gen 2 Minimal with Support by Rinne Labs: Rinne Labs offers a lightweight and secure Red Hat Enterprise Linux 9.4 image on a Generation 2 virtual machine. Gen 2 VMs on Azure use a UEFI-based boot architecture instead of BIOS-based architecture and have improved boot and installation times.
RigReports Drilling and Daily Operations Reporting – Pay-As-You-Go: RigReports from Mi4 is an oil and gas application that allows operators and consulting companies to track drilling data. RigReports integrates seamlessly with other applications to combine operations data with production or financial data.
Rocky Linux 8 CIS Level 1: This offer from Madarson IT provides a Rocky Linux 8 image hardened to meet Center for Internet Security (CIS) Level 1 compliance needs. CIS controls are internationally recognized cybersecurity best practices that map to many established standards and regulatory frameworks. Implementing CIS Level 1 benchmarks can help establish a baseline level of security and reduce risk exposure to common cyber threats.
Rocky Linux 8 CIS Level 2: This offer from Madarson IT provides a Rocky Linux 8 image hardened to meet Center for Internet Security (CIS) Level 2 compliance needs. Implementing CIS Level 2 benchmarks can help organizations improve their security posture and protect against advanced and persistent threats.
Rocky Linux 8.9 for Ansys by CIQ: This offer from CIQ provides Rocky Linux 8.9 along with NVIDIA drivers for Ansys. The stable, high-performance environment is tailored for Ansys Axis applications and is designed to enhance your engineering simulation workflows.
Rocky Linux 9 CIS Level 1: This offer from Madarson IT provides a Rocky Linux 9 image hardened to meet Center for Internet Security (CIS) Level 1 compliance needs. CIS benchmarks are internationally recognized as security standards for defending IT systems and data against cyberattacks.
Rocky Linux 9 CIS Level 2: This offer from Madarson IT provides a Rocky Linux 9 image hardened to meet Center for Internet Security (CIS) Level 2 compliance needs. Implementing CIS Level 2 benchmarks can help organizations improve their security posture and protect against advanced and persistent threats.
Rocky Linux 9.3: This offer from AskforCloud provides Rocky Linux 9.3 on a Microsoft Azure virtual machine. Rocky Linux is a community-driven enterprise-grade operating system that places significant focus on stability and performance. It can deliver a smooth transition for CentOS users through binary compatibility.
Rocky Linux 9.4: This offer from AskforCloud provides Rocky Linux 9.4 on a Microsoft Azure virtual machine. Rocky Linux is a community-driven enterprise-grade operating system that places significant focus on stability and performance. It can deliver a smooth transition for CentOS users through binary compatibility.
SERVICEME: GenAI Platform: SERVICEME from Shanghai Micronail Technology Co. Ltd. is a generative AI application center based on Microsoft Azure OpenAI Service. SERVICEME can be used to build AI copilots and agents for enterprises in manufacturing, retail, life sciences or other industries. These can improve efficiency in R&D, marketing, sales, customer service, and operations departments.
SoC Platform by INNOVATE: INNOVATE’s platform enhances detection, analysis, and response capabilities based on Microsoft Azure cloud services. The platform offers real-time security monitoring, threat detection, customizable reports, and automated incident alerts. This offer is available in Spanish.
Squid Auth Proxy Server on Red Hat 9: This offer from Tidal Media provides Squid along with Red Hat Enterprise Linux 9 on a Microsoft Azure virtual machine. Squid is an open-source web proxy server that secures internet connectivity, simplifies authentication, and offers comprehensive insights into internet usage.
Squid Protected Proxy Server on Red Hat 9: This offer from Tidal Media provides Squid along with Red Hat Enterprise Linux 9 on a Microsoft Azure virtual machine. Squid is an open-source web proxy server that offers advanced web content filtering, high-performance caching, granular access control policies, and real-time monitoring and reporting.
Teams Town Hall Pre-Event Validation: This service from Hive Streaming offers IT and network managers pre-event testing and assessments to identify weaknesses in their video delivery chain. Hive Streaming can provide detailed insights and recommendations to ensure a robust technical foundation for town-hall meetings in Microsoft Teams.
Teams Town Hall Real-Time Support: This service from Hive Streaming offers real-time monitoring and support for town-hall meetings in Microsoft Teams to ensure a smooth and secure experience. Users will receive training on real-time monitoring and mitigation systems, and there will be in-depth analysis and recommendations after their event.
Teams Town Hall Success Advisory: Get personalized technical consulting and success planning to implement town-hall meetings in Microsoft Teams. Hive Streaming can provide operational guidelines and discovery of primary use cases mapped to desired outcomes. This offer is intended for event producers, business stakeholders, and IT managers.
Ubuntu 18.04 with Jellyfin Server: This offer from Virtual Pulse S. R. O. provides a Jellyfin server along with Ubuntu 18.04 on a Microsoft Azure virtual machine. Jellyfin is open-source client-server software that centralizes your media library and enables remote access and sharing.
Ubuntu 22.04 with Minecraft Bedrock Game Server: This offer from Virtual Pulse S. R. O. provides a Minecraft: Bedrock Edition game server along with Ubuntu 22.04 on a Microsoft Azure virtual machine. Gamers can host their own server with full control over settings and configurations.
VPN Server: This virtual private network solution from IN4IT uses WireGuard technology and integrates with Azure Active Directory for secure access to Microsoft Azure. The client is easy to download and configure, making it convenient for developers needing internal network access.
WeTransact: Pax8 Offer: WeTransact simplifies the process of listing in the Microsoft commercial marketplace by handling technical aspects, ensuring partnership criteria are met, and offering guidance. SaaS companies can quickly publish and receive 1:1 direction and guidance.
Go further with workshops, proofs of concept, and implementations
Adastra’s Oracle Migration: Streamlining Transformation: Adastra offers a practical approach to transforming IT infrastructure. In this service, Adastra’s experienced team will assess your infrastructure, applications, databases, and data to identify areas for improvement and develop a migration strategy. Modernize your infrastructure and unlock new opportunities for growth.
AI Ideation with Microsoft Azure: 2-Day Workshop: In this workshop, Ailio GmbH will help businesses identify opportunities for AI implementation. Potential AI use cases will be prioritized based on value and complexity, and a lighthouse project will be chosen for further development. Participants will receive a detailed summary of workshop topics, along with a data evaluation and a strategy road map.
AI-Powered Spills Management: Cognizant’s AI-powered Spills Management solution uses the data management and AI capabilities of Microsoft Azure to improve reporting and predict spills for water utilities. The solution identifies outliers, flat lines, single spill events, and other anomalies, and it incorporates weather data and real-time sensor data to predict spills.
Azure Arc Enablement Solution: Adastra will deliver a two-hour “Art of the Possible” workshop focused on Microsoft Azure Arc, followed by a two-day discovery and design workshop to align an Azure Arc pilot program with your organization’s strategic objectives. The pilot program offers a unified control plane for Windows, Linux, SQL, VMWare, or containerized systems, and it includes onboarding for up to 25 servers to Azure Arc.
Azure Cloud Monthly Support Service: This managed service from OneNeck IT Solutions endeavors to enable clients to get the most out of their Microsoft Azure environments. It emphasizes cost optimization, incident response, and cloud adoption through the Azure Well-Architected Framework.
Azure Foundation for Enterprises: 3-Month Implementation: This comprehensive package from Public Cloud Group GmbH can serve as the basis for implementing Microsoft Azure in your enterprise. It encompasses governance, compliance, networking, deployment, automation, architecture, and organizational and technical readiness. This offer is available only in German.
Azure Foundation: 2-Day Proof of Concept: This workshop from Public Cloud Group GmbH will be delivered in three four-hour sessions, where participants will gain practical experience by building a proof-of-concept or demo environment that allows them to explore the features and capabilities of Azure. This offer is available only in German.
Azure Immersion Deep Dive: 5-Week Plan: Peritos Solutions offers a structured approach to implementing security measures and operational practices for DevOps services. Using Microsoft products such as Azure and Azure DevOps, Peritos Solutions will optimize your organization’s CI/CD pipelines and disaster recovery plans. Deliverables include ideation, design, building, testing, monitoring, customization, and training.
Azure Implementation: 2-Month Deployment: By enabling a strong and secure cloud foundation, your business can improve communication, security, reliability, and efficiency. In this engagement, RKON will deploy an Azure landing zone to establish the foundation for your migration. RKON understands the complex technical implications of migrating to the cloud and follows the structured Cloud Adoption Framework methodology.
Azure iPaaS Acceleration Pilot Program: Adastra helps businesses explore, plan, and implement solutions using Microsoft Azure Integration Platform as a Service. Adastra will start with a two-hour discovery workshop to discuss your business’s integration needs. This will be followed by a two days of design and planning for a customized iPaaS solution. An implementation will then commence over the next four weeks.
Azure Site Recovery Deployment: Microsoft Azure Site Recovery orchestrates and automates the replication of Azure virtual machines to keep your applications running during planned and unplanned outages. In this engagement, iNFINITUM will determine your business’s replication needs, design a solution, execute a proof of concept if necessary, and configure a production environment.
Bird Nest Detection: Celebal Technologies will implement its Bird Nest Detection solution, which uses Microsoft Azure services and advanced detection algorithms to identify bird nests in or near critical energy facilities, reducing the risk of equipment damage and downtime. Safeguard your energy infrastructure and reduce damage and outages by 50 percent.
Broken Shell Detection: A crucial facet of industrial maintenance is the identification and resolution of damaged or compromised equipment casings. Celebal Technologies will implement its Broken Shell Detection solution to safeguard your industrial equipment, minimize downtime, and optimize operations. Using Microsoft Azure services, the solution will detect fractures, corrosion, or other structural vulnerabilities.
Conversational AI Services: 6-Week Implementation: Conversational AI uses natural language processing and machine learning to create engaging, human-like conversations with customers. Popular channels for conversational AI include virtual assistants on websites, mobile and messaging apps, email automation, and voice assistants Softchoice will design and implement a conversational AI solution using Microsoft Azure OpenAI Service.
Data Democratization on Azure: 12-Week Implementation: Sigmoid will implement its Internal Data Marketplace to enable data democratization, making quality data available to all relevant stakeholders in your organization. Self-serve analytics capabilities will ensure data-driven decision making, fostering innovation and organizational growth.
DSC_Consulting: 3-Week Analysis and Workshop: Digital Survival Company offers consulting services to help customers migrate, modernize, and optimize applications and infrastructure on Microsoft Azure. In this analysis, Digital Survival Company will focus on your business objectives and how the cloud can contribute. Strategy will be matched to measurable goals and key performance indicators based on your architecture and security principles.
ELEKS: 2-Week Product Sprint Service: ELEKS helps businesses rapidly define and validate their product vision. In this sprint, ELEKS will suggest a technical approach and advise on the most cost-effective implementation method. Clients will receive a comprehensive digital application blueprint and a seamless transition to development work.
Energy Arbitrage: Celebal Technologies will provide an array of services to ensure successful implementation of its Energy Arbitrage solution, which maximizes energy market profits through smart electricity storage and trading. With real-time monitoring and intelligent algorithms, Energy Arbitrage lets you make the most of every energy unit, reducing costs during low-demand times and maximizing revenue during high-demand spikes.
Fabric Implementation: 8 Weeks: Microsoft Fabric is a modern data analytics platform that unifies and simplifies business data environments, providing built-in governance, security, and compliance. Insight will deliver a production-based Fabric environment, complete with light knowledge transfer, training, and documentation.
Fabric Proof of Concept: 4 Weeks: Insight will deliver a proof of concept of Microsoft Fabric to show how your organization can scale data and analytics. This will involve a current-state assessment, data ingestion pipeline creation, and a proprietary accelerating code set for faster delivery.
Gen AI: 5-Week Proof of Concept: In this proof of concept, iLink Systems will develop and validate a RAG-based (retrieval augmented generation) generative AI use case. iLink Systems will collaborate with you to identify use cases where RAG-based generative AI can deliver significant value by automating processes, enhancing decision-making, and driving innovation.
GenAI: 2-Day Workshop: This workshop from iLink Systems will help organizations identify and prioritize business use cases for AI. Participants will engage in interactive sessions to explore current business challenges and processes and map them to potential AI solutions. The workshop aims to provide a clear road map for integrating AI technologies to enhance operational efficiency, drive innovation, and achieve strategic goals.
Hourly Energy Matching & Allocation: In this engagement, Celebal Technologies will implement its Hourly Energy Matching & Allocation solution, which synchronizes electricity production and consumption on an hourly basis to optimize the use of energy resources. The solution eliminates wastage and mitigates the risk of overages, leading to cost savings and a more sustainable energy footprint.
Insulator Orientation Detection: Celebal Technologies will implement its Insulator Orientation Detection solution, which uses Microsoft Azure services to identify the correct positioning of insulators in electrical infrastructure. This proactive detection of potential faults or misalignments protects power transmission networks and keeps them operating at peak performance.
Marketing Measurement & Analytics: 6-Month Implementation: Sigmoid will implement a marketing measurement and analytics solution to enhance visibility into your company’s marketing performance on a weekly basis, which will allow you to optimize campaign strategy midflight. Taking into account your campaign goals, the solution can generate recommendations on your channel mix and spending.
Microsoft Azure AI Workshop: In this workshop, Sii’s solution architects and domain leaders will analyze your needs and determine how the introduction of Microsoft AI services can improve your business processes. A workshop summary will be provided that details the current situation, the solution vision, and the technology recommendations.
Microsoft OpenAI Workshop: In this workshop intended for solution architects, cloud practitioners, data scientists, and other business users, Protiviti will discuss Microsoft Fabric and generative AI uses cases available for plug-and-play support.
One-Day Innovation Workshop: This workshop from ELEKS will help businesses develop project plans. The workshop will involve examining problems, developing customer personas, and devising solutions. Clients will receive a high-level product concept and prioritized feature maps so they can move forward with their innovation initiatives.
Patient_360: 4-Week Proof of Concept: Harman Connected Services will deliver a proof of concept of its Intelligent Healthcare platform, which can integrate with its Patient Engagement platform and facilitate seamless data retrieval, remote monitoring, tailored medication adherence programs, and automated discharge summaries.
Set Up Business Recovery Plans in Azure: Blue Soft Empower, formerly known as Projetlys, will examine your IT infrastructure and applications to determine which are eligible for backup, then deliver a business recovery plan. This offer is available only in French.
Setup for Azure Landing Zone: 6-Week Implementation: Peritos Solutions simplifies cloud adoption for businesses of all sizes and industries. In this engagement, Peritos Solutions will establish an Azure landing zone tailored to meet the unique needs of your business.
Unlock Cloud Efficiency and Financial Control: Adastra’s FinOps service optimizes cloud usage and financial management through continuous calibration, cross-functional collaboration, financial control, and cloud migration opportunities. Adastra offers a structured and strategic approach to enhancing cloud financial management that enables cloud usage insights and cost optimization.
Unlock the Power of GitHub Copilot: In this remote workshop intended for developers, product managers, and technical subject matter experts, Apex Systems will show you the value of GitHub Copilot, which uses Azure infrastructure along with an AI-based vulnerability prevention system that blocks insecure coding patterns in real time. Learn how to harness this technology to drive improvements in your development processes.
Unlock Your Cloud Potential: Adastra’s Azure Modernization Assessment: Adastra’s assessment is designed to revamp your digital infrastructure for optimal performance and efficiency. The comprehensive package will delve deep into your current setup and future aspirations, delivering a customized road map for sustained growth and innovation.
Vegetation Encroachment Detection: Celebal Technologies will implement its Vegetation Encroachment Detection solution, which uses Microsoft Azure services to identify overgrown vegetation near power lines, mitigating potential disruptions and ensuring the uninterrupted flow of electricity. Safeguard your electrical grid and improve the reliability and efficiency of your energy infrastructure.
This article is contributed. See the original author and article here.
1. Background
In the rapidly evolving field of Generative AI, SLM/LLM fine-tuning and implementing RAG (Retrieval-Augmented Generation) techniques have become essential for achieving high-performance and domain-specific applications. Creating synthetic datasets for these purposes is crucial as it allows for the generation of tailored training data that addresses specific gaps and nuances in the target domain, which might not be adequately covered by existing datasets. This approach enhances the model’s ability to understand and generate relevant and accurate information, ultimately leading to more robust, reliable, and context-aware AI systems that can better serve users’ needs in specialized areas.
Generating high-quality datasets from diverse formats of raw data, such as PDFs, CSVs, and TXTs, especially those containing a mix of images, tables, and text, presents several significant challenges. This is mainly because the extraction process itself is complex, as each format requires different parsing techniques to accurately retrieve and interpret the content. PDFs, for instance, can have varied structures and may not follow a standardized layout, making it difficult to consistently extract text and images. Additionally, handling tables within PDFs is particularly challenging because they can span multiple pages and have complex cell structures.
If you want to improve the performance of your model using a seed dataset generated from raw data as a baseline, you may need data augmentation to generate high-quality synthetic data. But there is a risk of introducing biases or inconsistencies during the augmentation process. Augmented data needs to be representative of the diversity and variability present in the real-world data. If not carefully managed, augmentation can lead to overfitting, where the model performs well on augmented data but poorly on actual data, or it can amplify existing biases in the dataset. In this blog, we share in detail the methodology and code snippets that will help you solve these challenges.
2. Constructing a Seed Dataset
2.1. Overview
The task is to preprocess and convert this heterogeneous data into a structured format suitable for fine-tuning or RAG. This involves extracting and cleaning text from various file formats, converting tables and images to text using Azure AI Services if necessary. This dataset is used as a seed dataset for fine-tuning or RAG and serves as a baseline to improve the performance of domain-specific use cases. Here’s an easy to follow hands-on for you based on a typical use case. All of this code is uploaded here.
PDF
make_qa_multimodal_pdf_docai.ipynb: (Recommended) Generate QnA synthetic dataset from a Complex PDF using Azure AI Document Intelligence.
make_qa_multimodal_pdf_oss.ipynb: Generate QnA synthetic dataset from a Complex PDF using Open source (Unstructured toolkit for this hands-on). To run this file, you first need to install the required packages with startup_unstructured.sh. The installation will take a few minutes.
make_qa_only_image_pdf.ipynb: Generate QnA synthetic dataset from a PDF – Image-heavy PDF.
CSV
make_qa_csv.ipynb: This is the general case. It is not difficult to create a QnA dataset by reading and chunking with CSVLoader.
make_qa_image_url_csv.ipynb: This is another common case. If image url information is included, change this url to a summary result for that image.
Let’s take a look at the contents of the most representative make_qa_multimodal_pdf_docai.ipynb.
2.2. Separate the PDF pages
Separate the PDF pages into “Text”, “Image”, and “Mixed” with text, image, and table by using analyze_pdf_page_content(…)
def analyze_pdf_page_content(pdf_path, text_length_thres=600):
document = fitz.open(pdf_path)
page_analysis = defaultdict(list)
for page_num in range(len(document)):
page = document.load_page(page_num)
text = page.get_text("text")
image_list = page.get_images(full=True)
text_length = len(text)
num_images = len(image_list)
if text_length > text_length_thres and num_images == 0:
content_type = 'Text'
elif text_length <= text_length_thres and num_images > 0:
content_type = 'Image'
else:
content_type = 'Mixed'
page_analysis[content_type].append(page_num)
return dict(page_analysis)
Text-heavy page can be processed with open source (e.g., PyMuPDF) without the need to use toolkits like Azure AI Document Intelligence or Unstructured.
Image-heavy page can be converted the entire page to images and let a multimodal LLM like Azure OpenAI GPT-4o summarize each page.
Mixed page uses Azure AI Document Intelligence to separate images, text, and tables. Azure Document Intelligence’s built-in models offer key features for document analysis, including:
Text Extraction: Identifies and extracts text from various document types.
Table Recognition: Detects and extracts table structures.
Selection Marks: Recognizes checkboxes and radio buttons.
Form Field Recognition: Extracts fields from forms.
Document Structure Understanding: Differentiates between titles, headers, footers, and other sections.
Multi-language Support: Handles documents in multiple languages.
SDK and REST API Integration: Provides tools for seamless integration into applications.
Now let’s look at how to extract information from mixed pages in detail.
Extract mixed pages as prebuilt-layout models and convert them to markdown –document_intelligence_client.begin_analyze_document(“prebuilt-layout”, output_content_format=ContentFormat.MARKDOWN, …)
Extract images using bounding boxes (x, y, w, h) stored in figure tags – crop_image_from_file(…). If the bounding box size is too small (is_bounding_box_larger_than(…)) or the image is a simple pattern with no meaning (image_complexity(…)), the image is not extracted.
Summarize the extracted images with GPT-4o. (understand_image_with_gpt(…))
Here is a code snippet that summarizes this briefly. Note that the actual implementation is more complex than this.
# Import necessary functions for processing
from util.preprocess import (
image_complexity, is_bounding_box_larger_than, crop_image_from_file,
understand_image_with_gpt, update_figure_description
)
if "Mixed" in analyzed_pdf_result:
pdf_mixed_path = path_to_mixed_pdf
# Open and analyze the PDF file with Azure Document Intelligence
with open(pdf_mixed_path, "rb") as file:
poller = document_intelligence_client.begin_analyze_document(
"prebuilt-layout", analyze_request=file, content_type="application/octet-stream",
output_content_format=ContentFormat.MARKDOWN
)
result = poller.result()
md_content = result.content
output_folder = "pdf_mixed_tmp"
delete_folder_and_make_folder(output_folder)
input_file_path = file_path
if result.figures:
for idx, figure in enumerate(result.figures):
figure_content = ""
img_description = ""
# Extract figure content from markdown
for span in figure.spans:
figure_content += md_content[span.offset:span.offset + span.length]
for region in figure.bounding_regions:
boundingbox = extract_bounding_box(region)
# Extract images only when the bounding box size is greater than a certain size.
if is_bounding_box_larger_than(boundingbox):
cropped_image = crop_image_from_file(input_file_path, region.page_number - 1, boundingbox)
# Extract images only when the image complexity is high.
if image_complexity(cropped_image) == "Complex":
cropped_image_filename = save_cropped_image(cropped_image, idx, input_file_path, output_folder)
try:
image_summarization = understand_image_with_gpt(client, aoai_deployment_name, cropped_image_filename, "", max_tokens, language)
except openai.BadRequestError:
image_summarization = ""
img_description += image_summarization
# Update the figure tag in the extracted markdown document
md_content = update_figure_description(md_content, img_description, idx)
image_complexity(…) assesses the complexity of an image by analyzing its histogram entropy, Laplacian variance, and edge count. It converts the image to a format suitable for OpenCV processing, calculates the entropy of the color histograms, the variance of the Laplacian to measure focus, and the number of edges detected by the Canny edge detector. Based on these metrics, it classifies the image as either “Complex” or “Simple” depending on predefined threshold values for each metric. The code snippet is below.
We can leverage the azure-ai-generative package. The QADataGenerator class in this package makes it easy to generate QnA synthetic questions. However, using this class as is has the disadvantage of not being able to use custom prompts, so we inherited from it and created the CustomQADataGenerator class as follows.
All you have to do is put your own prompts into a text file and you’re done. There are some prompt examples at this link.
Now, you can easily create a QnA dataset for fine-tuning using the code snippet below. Of course, you can also use it for RAG with just a little modification of the code.
import os
import asyncio
from collections import Counter
from typing import Dict
from azure.ai.generative.synthetic.qa import QAType
from util.qa import CustomQADataGenerator
model_config = {
"deployment": os.getenv("AZURE_OPENAI_DEPLOYMENT_NAME"),
"model": "gpt-4o",
"max_tokens": 2000,
}
qa_generator = CustomQADataGenerator(model_config=model_config, templates_dir=f"./prompt_template/{LANGUAGE_CODE}")
concurrency = 6 # number of concurrent calls
sem = asyncio.Semaphore(concurrency)
#qa_type = QAType.CONVERSATION
qa_type = QAType.LONG_ANSWER
async def generate_async(text: str) -> Dict:
async with sem:
return await qa_generator.generate_async(
text=text,
qa_type=qa_type,
num_questions=3, # Number of questions to generate per text
)
input_batch = mixed_chunks + text_chunks + image_summaries
results = await asyncio.gather(*[generate_async(text) for text in input_batch], return_exceptions=True)
question_answer_list = []
token_usage = Counter()
for result in results:
if isinstance(result, Exception):
raise result # exception raised inside generate_async()
question_answer_list.append(result["question_answers"])
token_usage += result["token_usage"]
print("Successfully generated QAs")
The screenshot below is a Q&A result extracted from sample raw data, you can see sample results in this folder, all of the sample raw data is based on articles I have written or data I have generated, so there are no license issues. If you are doing a PoC/MVP, please prepare your own dataset.
Below is a comparison of the results before and after fine tuning of GPT-4o without RAG for a Korea customer PoC. GPT-4o is available to a small number of customers as a private preview as of July 2024. This is the result of creating a set of 16 questions and answers for PoC and comparing three indicators of Similarity, Coherence, and Fluency in Azure AI studio. The values of the indicator are on a scale of 1-5, with higher values being better.
3. Synthetic Data Generation
After fine-tuning with the generated dataset from the above section, a baseline was established, but the performance requires improvement due to a lack of data (e.g., there are only 1,000 samples in the dataset). In this case, a synthetic dataset must be created by applying data augmentation techniques to enhance performance. The data augmentation technique utilizes the representative techniques announced by Microsoft: Evol-Instruct, GLAN (Generalized Instruction Tuning), and Auto Evol-Instruct.
Evol-Instruct: Generate diverse instructional data to augment the dataset from the seed datset.
GLAN: Apply generalized instruction tuning to expand the variety of Q&A pairs.
Auto Evol-Instruct: Automate the generation of synthetic data to scale the augmentation process.
3.1. Augment your dataset – Evol-Instruct
The Evol-Instruct concept developed by Microsoft aims to enhance the capabilities of LLMs by automatically evolving instructions to various complexity levels, instead of relying solely on manually created instructions. This method involves several key components and steps:
Instruction Evolution: Starting with an initial set of instructions, the model uses a LLM like GPT-4o to iteratively rewrite these instructions into more complex versions. The evolution process involves two types of instruction enhancement: in-depth evolving (adding constraints, deepening, increasing reasoning steps) and in-breadth evolving (generating new, diverse instructions).
Response Generation: After evolving the instructions, the LLM generates responses to these newly complex instructions, ensuring that they are reasonable and understandable by humans.
Instruction Elimination: The process includes a step to filter out any instructions that fail to evolve properly, ensuring only high-quality, challenging instructions remain.
This open-source implementation is based on the WizardLM paper and h2o-wizardlm. We added the following features to the original implementation:
Modified it to be able to call Azure OpenAI by adding the AzureGPTPipeline class.
The prompt has been refined and modified to support multiple languages. Use -language argument for other language. (e.g., -language Korean)
Made it possible to create questions only when necessary. A better strategy is to create questions and answers separately. Use -question_only argument. (e.g., -questioin_only True)
Prevented infinite loop. mutate() in the original implementation determines the validity of the augmented statement and repeats the loop until it is valid. However, this process takes a very long time and there is a problem in that the loop repeats infinitely in certain situations.
You can easily convert your jsonl file from the previous section with convert.py and augment your dataset with evolve.py.
{"idx": 1, "Skill": "Distributed training on Cloud", "Difficulty": 5, "Instruction": "What version of TensorFlow was used in the evaluation?"}
{"idx": 2, "Skill": "Distributed training on Cloud", "Difficulty": 5, "Instruction": "What is the first step to prepare the validation set for ImageNet training?"}
{"idx": 3, "Skill": "Distributed training on Cloud", "Difficulty": 5, "Instruction": "What is the purpose of the script 'preprocess_imagenet.py' and how is it executed?"}
Synthetic instruction – after running evolve.py (shows only 10 samples)
[
{
"input": "Could you determine the specific learning rate value (LR) recorded at the 7000th iteration within the hvd_train_log?"
},
{
"input": "Can you provide the accuracy and precision metrics recorded at epoch 37 in the model training logs?"
},
{
"input": "Could you elaborate on the process and specific steps needed to resize the TFRecord training dataset for ImageNet utilizing the provided scripts, including any potential challenges and solutions?"
},
{
"input": "Could you outline the sequential procedures to be followed post-data transformation to ensure both the reliability of data backup and its continuous availability, requiring detailed verification and contingency planning?"
},
{
"input": "Could you explain the primary function and broader implications of using the 'nohup' command, especially in the context of manually downloading the extensive ImageNet dataset, along with potential advantages and disadvantages?"
},
{
"input": "What is the purpose of using the 'nohup' command when manually downloading the ImageNet dataset?"
},
{
"input": "Can you explain the rationale behind not performing the resizing to 224x224 for the RecordIO format, and discuss any potential consequences and alternatives?"
},
{
"input": "What are the underlying reasons that make a supplementary EBS volume of 1.0TB advisable for efficiently handling the ImageNet dataset? Please elaborate on the factors involved."
},
{
"input": "Taking into account both cost-efficiency and performance, what is the recommended EC2 instance type for storing the ImageNet dataset and why?"
},
{
"input": "What is the purpose of the script 'preprocess_imagenet.py' and how is it executed?"
}
]
Example datasets are placed in this folder. Please try the minimal example first and configure your dataset by referring to the tunable parameters.
3.2. Improve the generalizability of your model – GLAN
Catastrophic forgetting, also known as catastrophic interference, occurs during SLM/LLM fine-tuning when a model trained on new data overwrites the knowledge it previously acquired, leading to a significant drop in performance on earlier tasks. This issue is particularly prominent in scenarios where the model needs to adapt to diverse and potentially conflicting data distributions without losing its ability to perform well on initial tasks.
GLAN (Generalized Instruction Tuning) addresses catastrophic forgetting by leveraging a systematically curated taxonomy of human knowledge to generate synthetic, diverse, and comprehensive instruction datasets. Specifically, GLAN uses a pre-curated taxonomy of human knowledge and capabilities to generate large-scale, diverse instruction data across various disciplines. This taxonomy mirrors the systematic structure of human education, breaking down knowledge into fields, sub-fields, and distinct disciplines, which are then used to design syllabi for different subjects. These syllabi detail key concepts for each class session, enabling the generation of diverse and comprehensive instructions. Here is GLAN’s key features:
Taxonomy Creation: A detailed hierarchy of human knowledge is built using LLMs and human verification.
Subject and Syllabus Generation: LLMs generate a list of subjects and detailed syllabi, breaking down subjects into class sessions with key concepts.
Instruction Generation: Homework questions and their answers are generated based on the syllabi, ensuring broad coverage and diversity.
The author has implemented the concept of this paper from the scratch and made it public. Let’s take a look at the core concepts of the code.
generate_taxonomy(…): Generate a taxonomy of human knowledge and capabilities.
generate_subjects(…): Generate a list of subjects for a given discipline. Please refer to section 2.2 of the paper.
generate_syllabus(…): Generate a syllabus for a given subject at a specific level. Please refer to section 2.3 of the paper.
sample_class_sessions_and_key_concepts(…): Sample class sessions and key concepts to generate questions of varying difficulty.
generate_questions(…): Generate questions based on class sessions and key concepts using LangChain pipeline. Please refer to section 2.4 of the paper.
generate_answers(…): Generate answers to the questions using LangChain pipeline. Please refer to section 2.4 of the paper.
glan_instruction_generation(…): GLAN pipeline
This pseudocode for glan_instruction_generation(…) outlines the steps of generating or loading disciplines, generating subjects and syllabi for each discipline, creating questions based on the syllabi, saving the questions, and optionally generating and saving answers to create a comprehensive instruction dataset.
This implementation supports all languages supported by LLM, so you can easily create datasets in your country’s language. Below is an example created with this code.
English language
Korean language
Example datasets are placed in this folder. Please try the minimal example first and configure your dataset by referring to the tunable parameters.
The code snippet below is a shell script that generates a large synthetic dataset. If you are willing to pay enough PTUs, you can create your own datasets just by running this script.
# Initialize counter
counter=1
# Read disciplines.txt line by line
while IFS= read -r line || [[ -n "$line" ]]; do
# Create the corresponding disciplines file
discipline_file="disciplines_line${counter}.txt"
echo Created "$discipline_file"
echo "$line" > "$discipline_file"
# # Run the Python script with the current disciplines file
python generate.py
--disciplines_filepath "$discipline_file"
--language Korean
--max_number_of_subjects 15
--max_number_of_subtopics 30
--max_number_of_session_name 30
--num_iterations 15
--num_questions_per_iteration 18
--question_max_tokens 1024
--question_batch_size 9
--model_name_for_answer gpt-4o
--answer_max_tokens 2048
--answer_batch_size 9
# Increment counter
((counter++))
# Delete the temporary disciplines file
rm "$discipline_file"
done < disciplines_sample.txt
The author has uploaded a synthetic dataset in Korean to the Hugging Face Hub under a public license. Korean customers are welcome to use this dataset as a baseline!
In the realm of Generative AI, creating and augmenting datasets for fine-tuning SLM/LLM models is a crucial step to ensure robust, reliable, and context-aware AI systems. This blog post outlined the process of constructing a seed dataset from diverse raw data formats and then applying data augmentation techniques such as Evol-Instruct and GLAN to enhance the dataset’s quality and diversity. These techniques help mitigate the challenges of extracting high-quality data from complex formats like PDFs and CSVs and prevent issues such as catastrophic forgetting during the fine-tuning process.
By leveraging advanced methods like GLAN, we can systematically generate comprehensive instruction datasets across various disciplines, enhancing the model’s performance without compromising previously learned knowledge. This approach not only improves the generalizability of the model but also ensures that it can handle a wide range of complex tasks effectively.
For those interested in exploring these techniques further and implementing them in their projects, all the code and examples discussed in this blog are available on the Azure synthetic-qa-generation GitHub repository. This resource provides a practical guide and tools to generate high-quality QnA datasets, enabling you to fine-tune and optimize your AI models for specialized applications. By following the methodologies and utilizing the tools provided, developers and data scientists can create robust datasets that significantly enhance the capabilities of their AI models, paving the way for more advanced and reliable AI solutions in various domains.
This article is contributed. See the original author and article here.
Problem
LinkedIn offers a valuable feature that allows users to create and download resumes directly from their profiles, effectively eliminating the challenges associated with resume formatting. However, students, being inexperienced, often struggle to craft high-quality resumes. As one of the career mentors, I find myself reviewing over 200 student resumes in an iterative process. Unfortunately, due to the sheer volume, my colleagues often overlook the quality of these resumes, allowing students to indiscriminately send out subpar or error-ridden resumes to potential employers.
This practice has resulted in a decreased employment rate and has negatively impacted the reputation of our course.
Furthermore, career mentors need to review/check each resume, analysis student’s profile, provide feedback to student and refer them to different types of job role.
Solution
We request students to upload their LinkedIn Resume PDFs to our Learning Management System (LMS) – Moodle as a part of their assignment. We frequently review these resumes using Azure OpenAI ChatGPT-4o.
In this post, I won’t delve into the specifics of data preprocessing, but here are the key steps involved:
Unzip the submitted resumes.
Rename the folder to the respective student’s name, ensuring there are no duplicates.
Transform each page of the LinkedIn PDF resume into a PNG format.
AI Resume Reviewer
AI career master’s system prompt, and it shows the function for AI Resume Reviewer.
As a dedicated career guide, your responsibility is to meticulously examine student resumes and provide feedback in Markdown format. Here are the detailed instructions:
Identify and enumerate contact details, list actual value of the email address, mobile number, and LinkedIn Profile URL, in the initial section.
List out all URLs present in the resume.
List out all technologies mentioned.
List out all skills highlighted.
List out all certifications acquired.
List out all educational qualifications along with the duration.
List out all professional experiences along with the duration.
The resume **should** contain an email and phone number for communication. Issue an alert if these details are missing.
The profile section **should** contain the student’s name, course name, institution, and GitHub URL. Issue an alert if any of these elements are missing.
Students are anticipated to be enrolled in the **Higher Diploma in Cloud and Data Centre Administration** course in Hong Kong. Issue an alert if this information is missing or incorrect.
Be vigilant for any illogical content (excluding irrelevant/non-IT work experience) or spelling mistakes. Issue an alert and underline the errors if any are detected.
The summary section should be devoid of any pronouns.
Ensure the consistency of tenses throughout the resume.
Propose a suitable job title for the student based on the resume content.
Assign a “Resume Rating” on a scale of 1 to 10, where 10 signifies an outstanding resume.
If there are any alerts or missing information, the “Resume Rating” **should not** exceed 5.
If the phone number or email address is missing, the “Resume Rating” **should** be 0.
Assume the role of an IT interviewer and justify the “Resume Rating”, correlating it with the likelihood of securing a job.
Suggest the kind of job the student is likely to land, such as a Cloud Engineer, Data Centre Technician, or Network Engineer, based on the resume content.
Group resumes images for each student
import os
from collections import defaultdict
# Define the path to the "data" folder
data_folder = "data"
cv_images = []
# Traverse through each subfolder inside the "data" folder
for root, dirs, files in os.walk(data_folder):
# Iterate over each file in the current subfolder
for file in files:
# Check if the file has a PNG extension
if file.endswith(".png"):
# Print the file path
# print(os.path.join(root, file))
cv_images.append(os.path.join(root, file))
# Group cv_images by folder
cv_images_by_folder = defaultdict(list)
for image_path in cv_images:
folder = os.path.dirname(image_path)
cv_images_by_folder[folder].append(image_path)
Prepare the chat prompts
import base64
# Function to encode an image file as a base64 string
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
# Function to create messages for the AI model
def create_messages(base64_images):
return [
{"role": "system", "content": system_prompt},
{"role": "user", "content": [
{"type": "text", "text": "Describe the images as an alternative text, provide feedback, warning if any and ratiing on the resume."},
*[
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img}"}}
for img in base64_images
]
]}
]
AI Review and saves the result for each student
from tqdm import tqdm
import os
from langchain_openai import AzureChatOpenAI
llm = AzureChatOpenAI(
openai_api_version=os.getenv("AZURE_OPENAI_GPT4O_API_VERSION"),
azure_deployment=os.getenv("AZURE_OPENAI_GPT4O_DEPLOYMENT_NAME"),
temperature=0,
)
# Sort the cv_images_by_folder dictionary by folder
sorted_cv_images_by_folder = dict(sorted(cv_images_by_folder.items(), key=lambda x: x[0]))
for folder, images in tqdm(sorted_cv_images_by_folder.items(), desc="Processing folders"):
save_path = os.path.join(folder, 'chatgpt_result.md')
if os.path.exists(save_path):
continue
encode_images = [encode_image(image) for image in images]
messages = create_messages(encode_images)
ai_message = llm.invoke(messages)
# print(ai_message.content)
# Save ai_message.content to a file
with open(save_path, 'w') as file:
file.write(ai_message.content)
Masked sample results 1
### Alternative Text Description
The image is a resume for XXXXXXXX. The resume is divided into three main sections: Contact, Experience, and Education.
– Higher Diploma in Cloud and Data Centre Administration, Cloud Computing (2023 – 2025)
#### Professional Experiences
– Casual Sales Assistant at DFI Retail Group (August 2023 – Present, 11 months)
### Alerts and Warnings
1. **Missing Technologies and Skills:** The resume does not mention any specific technologies or skills.
2. **Missing Certifications:** No certifications are listed.
3. **Profile Section:** The profile section is missing the GitHub URL.
4. **Course Information:** The course name and institution are correctly mentioned.
5. **Spelling and Grammar:** No spelling mistakes detected.
6. **Summary Section:** The summary section is devoid of pronouns.
7. **Tense Consistency:** The tenses are consistent throughout the resume.
### Suggested Job Title
– Entry-Level Cloud and Data Centre Technician
### Resume Rating
**Rating: 4/10**
### Justification
The resume contains the essential contact details and educational qualifications, which are crucial for any job application. However, it lacks specific technologies, skills, and certifications that are vital for a career in cloud and data centre administration. The absence of a GitHub URL in the profile section is also a significant omission. These missing elements reduce the likelihood of securing a job in the desired field.
### Suggested Job
Based on the current content of the resume, the student is likely to land an entry-level position such as a Cloud and Data Centre Technician. To improve the chances of securing a job, it is recommended to include relevant technologies, skills, and certifications.
Masked sample results 2
### Alternative Text Description
The image is a resume for Kelvin Yiu, an XYZ Cloud Club Captain from New Territories, Hong Kong SAR. The resume is divided into several sections: Contact, Top Skills, Languages, Certifications, Summary, Experience, and Education.
– XYZ Academy Graduate – XYZ Academy Cloud Foundations
#### Summary
A tech enthusiast with at least 3 years of hands-on experience in developing with Python and Golang, working on several cloud projects. Has a cybersecurity background and led a team to participate in numerous public cybersecurity competitions in Hong Kong during high school studies.
#### Experience
**Amazon Web Services (XYZ)**
– **Role:** Cloud Captain
– **Duration:** March 2024 – Present (3 months)
– **Location:** Hong Kong SAR
– **Responsibilities:**
– Started the first XYZ Cloud Club in Hong Kong.
– Planned events to teach about clouds and prepare people for jobs in cloud technology.
– Helped students join XYZ Cloud Clubs to build a cloud community.
– Led the growth of the Hong Kong Regional Cloud Club.
#### Education
**Hong Kong Institute of Information Technology (HKIIT) at IVE (Lee Wai Lee)**
– **Course:** Higher Diploma in Cloud and Data Centre Administration, Cloud Computing
– Contains the pronoun “I” which should be avoided.
– Spelling mistake: “I have” should be “I have”.
3. **Course Information:**
– Correct course information is present.
### Resume Rating
**Rating: 4/10**
### Justification
The resume contains essential contact details, educational qualifications, and professional experiences. However, it has several issues:
– The summary section contains a pronoun and a spelling mistake.
– The GitHub URL is missing from the profile section.
– The professional experience is relatively short (3 months).
These issues reduce the overall quality and effectiveness of the resume, making it less likely to secure a job.
### Suggested Job Title
– Cloud Engineer
– Data Centre Technician
### Likely Job
Based on the resume content, the student is likely to land a job as a Cloud Engineer or Data Centre Technician.
AI Resume Extractor
It retrieves all the review outcomes and exports them to a Microsoft Excel file. This process involves a function calling that guarantees the data is returned in the correct format and mapped to a structured record.
Get all AI reviews
import os
# Define the path to the "data" folder
data_folder = "data"
chatgpt_results = []
# Traverse through each subfolder inside the "data" folder
for root, dirs, files in os.walk(data_folder):
# Iterate over each file in the current subfolder
for file in files:
if file == "chatgpt_result.md":
# Print the file path
chatgpt_results.append(os.path.join(root, file))
chatgpt_results.sort()
Setup the function calling with LangChain and Pydantic.
from langchain_core.utils.function_calling import convert_to_openai_function
from typing import List, Optional
from langchain.pydantic_v1 import BaseModel, Field
class StudentCvRecord(BaseModel):
"""Call this to save a student CV record in markdown format."""
name: str = Field(description="Name of the student")
email: Optional[str] = Field(description="Email address")
mobile_number: Optional[str] = Field(description="Contact number")
linkedin_profile_url: str = Field(description="LinkedIn profile url")
resume_rating: int = Field(
description="Rating of the resume between 1 to 10")
rationale: str = Field(description="Rationale for the rating")
warning: str = Field(description="Any warning message")
feedback: str = Field(description="Feedback message")
proposed_job_titles: List[str] = Field(description="Proposed job titles")
certifications: List[str] = Field(description="List of certifications")
technologies: List[str] = Field(description="List of technologies")
skills: List[str] = Field(description="List of skills")
work_experience: List[str] = Field(description="List of work experiences")
student_cv_record_function = convert_to_openai_function(StudentCvRecord)
Extract the result for each student, and fallback to GPT-4o if GPT-35-turbo cannot handle the JSON encode.
import json
from tqdm import tqdm
student_records = []
for result_path in tqdm(chatgpt_results):
result_path_json = result_path.replace(".md", ".json")
if os.path.exists(result_path_json):
with open(result_path_json, "r") as f:
result_json = f.read()
result = StudentCvRecord.parse_raw(result_json)
student_records.append(result)
continue
with open(result_path, "r") as f:
cv = f.read()
name = result_path.split("/")[-2]
try:
result = chain35.invoke({"cv": cv})
except Exception as e:
result = chain4o.invoke({"cv": cv})
result.name = name
result_json = json.dumps(result.dict())
with open(result_path_json, "w") as f:
f.write(result_json)
student_records.append(result)
Microsoft Excel Report
Now, we can mail merge the result to students and let them fix their resumes.
Create data folder and upload zipped PDF resumes in it.
Modify zip_file_path and run data-preprocessing.ipynb
Run ai-resume-reviewer.ipynb to use Azure OpenAI ChatGPT4o to review resumes images.
Run ai-resume-extractor.ipynb to use Azure OpenAI ChatGPT 3.5 Tubo and 4o to extract the reviewer result.
Conclusion
The integration of Azure OpenAI ChatGPT-4o into our resume review process has significantly improved the quality of student resumes. By automating the initial review and feedback process, we ensure that each resume is meticulously examined for errors, missing information, and overall quality. This approach not only saves time for career mentors but also enhances the employability of our students by providing them with high-quality resumes. As a result, we have observed an increase in employment rates and a positive impact on the reputation of our course. This innovative solution demonstrates the potential of AI in transforming educational and career support services.
Enhancing a LinkedIn-generated resume PDF encourages students to maintain an impressive LinkedIn online presence. It’s crucial to uphold a well-crafted LinkedIn profile throughout one’s career.
DrWare.com uses cookies to provide you the best possible experience. Use of this site means you understand and accept our cookie policy. Find out more.
Recent Comments