by Contributed | Jun 3, 2024 | Technology
This article is contributed. See the original author and article here.
Introduction
In this article we will demonstrate how we leverage GPT-4o capabilities, using images with function calling to unlock multimodal use cases.
We will simulate a package routing service that routes packages based on the shipping label using OCR with GPT-4o.
The model will identify the appropriate function to call based on the image analysis and the predefined actions for routing to the appropriate continent.
Background
The new GPT-4o (“o” for “omni”) can reason across audio, vision, and text in real time.
- It can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in a conversation.
- It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50% cheaper in the API.
- GPT-4o is especially better at vision and audio understanding compared to existing models.
- GPT-4o now enables function calling.
The application
We will run a Jupyter notebook that connects to GPT-4o to sort packages based on the printed labels with the shipping address.
Here are some sample labels we will be using GPT-4o for OCR to get the country this is being shipped to and GPT-4o functions to route the packages.
The environment
The code can be found here – Azure OpenAI code examples
Make sure you create your python virtual environment and fill the environment variables as stated in the README.md file.
The code
Connecting to Azure OpenAI GPT-4o deployment.
from dotenv import load_dotenv
from IPython.display import display, HTML, Image
import os
from openai import AzureOpenAI
import json
load_dotenv()
GPT4o_API_KEY = os.getenv("GPT4o_API_KEY")
GPT4o_DEPLOYMENT_ENDPOINT = os.getenv("GPT4o_DEPLOYMENT_ENDPOINT")
GPT4o_DEPLOYMENT_NAME = os.getenv("GPT4o_DEPLOYMENT_NAME")
client = AzureOpenAI(
azure_endpoint = GPT4o_DEPLOYMENT_ENDPOINT,
api_key=GPT4o_API_KEY,
api_version="2024-02-01"
)
Defining the functions to be called after GPT-4o answers.
# Defining the functions - in this case a toy example of a shipping function
def ship_to_Oceania(location):
return f"Shipping to Oceania based on location {location}"
def ship_to_Europe(location):
return f"Shipping to Europe based on location {location}"
def ship_to_US(location):
return f"Shipping to Americas based on location {location}"
Defining the available functions to be called to send to GPT-4o.
It is very IMPORTANT to send the function’s and parameters descriptions so GPT-4o will know which method to call.
tools = [
{
"type": "function",
"function": {
"name": "ship_to_Oceania",
"description": "Shipping the parcel to any country in Oceania",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The country to ship the parcel to.",
}
},
"required": ["location"],
},
},
},
{
"type": "function",
"function": {
"name": "ship_to_Europe",
"description": "Shipping the parcel to any country in Europe",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The country to ship the parcel to.",
}
},
"required": ["location"],
},
},
},
{
"type": "function",
"function": {
"name": "ship_to_US",
"description": "Shipping the parcel to any country in the United States",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The country to ship the parcel to.",
}
},
"required": ["location"],
},
},
},
]
available_functions = {
"ship_to_Oceania": ship_to_Oceania,
"ship_to_Europe": ship_to_Europe,
"ship_to_US": ship_to_US,
}
Function to base64 encode our images, this is the format accepted by GPT-4o.
# Encoding the images to send to GPT-4-O
import base64
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
The method to call GPT-4o.
Notice below that we send the parameter “tools” with the JSON describing the functions to be called.
def call_OpenAI(messages, tools, available_functions):
# Step 1: send the prompt and available functions to GPT
response = client.chat.completions.create(
model=GPT4o_DEPLOYMENT_NAME,
messages=messages,
tools=tools,
tool_choice="auto",
)
response_message = response.choices[0].message
# Step 2: check if GPT wanted to call a function
if response_message.tool_calls:
print("Recommended Function call:")
print(response_message.tool_calls[0])
print()
# Step 3: call the function
# Note: the JSON response may not always be valid; be sure to handle errors
function_name = response_message.tool_calls[0].function.name
# verify function exists
if function_name not in available_functions:
return "Function " + function_name + " does not exist"
function_to_call = available_functions[function_name]
# verify function has correct number of arguments
function_args = json.loads(response_message.tool_calls[0].function.arguments)
if check_args(function_to_call, function_args) is False:
return "Invalid number of arguments for function: " + function_name
# call the function
function_response = function_to_call(**function_args)
print("Output of function call:")
print(function_response)
print()
Please note that WE and not GPT-4o call the methods in our code based on the answer by GTP4-o.
# call the function
function_response = function_to_call(**function_args)
Iterate through all the images in the folder.
Notice the system prompt where we ask GPT-4o what we need it to do, sort labels for packages routing calling functions.
# iterate through all the images in the data folder
import os
data_folder = "./data"
for image in os.listdir(data_folder):
if image.endswith(".png"):
IMAGE_PATH = os.path.join(data_folder, image)
base64_image = encode_image(IMAGE_PATH)
display(Image(IMAGE_PATH))
messages = [
{"role": "system", "content": "You are a customer service assistant for a delivery service, equipped to analyze images of package labels. Based on the country to ship the package to, you must always ship to the corresponding continent. You must always use tools!"},
{"role": "user", "content": [
{"type": "image_url", "image_url": {
"url": f"data:image/png;base64,{base64_image}"}
}
]}
]
call_OpenAI(messages, tools, available_functions)
Let’s run our notebook!!!
Running our code for the label above produces the following output:
Recommended Function call:
ChatCompletionMessageToolCall(id='call_lH2G1bh2j1IfBRzZcw84wg0x', function=Function(arguments='{"location":"United States"}', name='ship_to_US'), type='function')
Output of function call:
Shipping to Americas based on location United States
That’s all folks!
Thanks
Denise
by Contributed | Jun 2, 2024 | Technology
This article is contributed. See the original author and article here.
Spotlight on AI in your DevOps Lifecycle
Explore the transformative power of artificial intelligence in DevOps with our comprehensive series, “Spotlight on AI in Your DevOps Lifecycle.” This series delves into the integration of AI into every stage of the DevOps process, providing invaluable insights and practical guidance. Whether you’re a seasoned professional or new to the field, these episodes will equip you with the knowledge to leverage AI effectively in your development and operations lifecycle.
Speakers
Sessions: Register Now. https://aka.ms/DevOpsAISeries
DevOps in the era of Generative AI: Foundations of LLMOps
With the advent of generative AI, the development life cycle of intelligent applications has undergone a significant change. This shift from classical ML to LLMs-based solutions leads to implications not only on how we build applications but also in how we test, evaluate, deploy, and monitor them. The introduction of LLMOps is an important development that requires understanding the foundations of this new approach to DevOps.
The session “DevOps in the era of Generative AI: Foundations of LLMOps” will explore the basics of LLMOps, providing examples of tools and practices available in the Azure ecosystem. This talk will be held on June 12th, 2024, from 4:00 PM to 5:00 PM (UTC).
Register Now. https://aka.ms/DevOpsAISeries
Continuous Integration and Continuous Delivery (CI/CD) for AI
The session “Continuous Integration and Continuous Delivery (CI/CD) for AI” will focus on MLOps for machine learning and AI projects. This talk will cover how to set up CI/CD and collaborate with others using GitHub. It will also discuss version control, automated testing, and deployment strategies.
The session will take place on June 20th, 2024, from 6:00 PM to 7:00 PM (UTC).
Register Now. https://aka.ms/DevOpsAISeries
Monitoring, Logging, and AI Model Performance
Building an AI application does not stop at deployment. The core of any AI application is the AI model that performs certain tasks and provides predictions to users. However, AI models and their responses change over time, and our applications need to adapt to these changes in a scalable and automated way.
The session “Monitoring, Logging, and AI Model Performance” will explore how to use tools to monitor the performance of AI models and adapt to changes in a scalable way. This talk will be held on June 26th, 2024, from 4:00 PM to 5:00 PM (UTC).
Register Now. https://aka.ms/DevOpsAISeries
Scaling and Maintaining Your Applications on Azure
Azure is a popular cloud platform that provides many benefits for running AI applications. This session will focus on the practical aspects of running your applications on Azure, with a special emphasis on leveraging Azure OpenAI and Python FastAPI. The talk will cover best practices for scaling your applications to meet demand and maintaining their health and performance.
The session will be held on July 3rd, 2024, from 4:00 PM to 5:00 PM (UTC).
Register Now. https://aka.ms/DevOpsAISeries
Security, Ethics, and Governance in AI
AI brings many exciting new features into the tech landscape, but it also introduces new security risks and challenges. In this session, we will learn about the best practices and tools for securing AI-enabled applications and addressing ethical and governance issues related to AI.
The session will take place on July 10th, 2024, from 4:00 PM to 5:00 PM (UTC).
Register Now. https://aka.ms/DevOpsAISeries
by Contributed | Jun 1, 2024 | Technology
This article is contributed. See the original author and article here.
Navigating the Future with Microsoft Copilot: A Guide for Technical Students
Introduction
Copilot learning hub
Copilot is an AI assistant powered by language models, which offers innovative solutions across the Microsoft Cloud. Find what you, a technical professional, need to enhance your productivity, creativity, and data accessibility, and make the most of the enterprise-grade data security and privacy features for your organization.
As a technical student, you’re always on the lookout for tools that can enhance your productivity and creativity.
Enter Microsoft Copilot, your AI-powered assistant that’s revolutionizing the way we interact with technology. In this blog post, we’ll explore how Copilot can be a game-changer for your learning and development.
Understanding Copilot Microsoft Copilot is more than just an AI assistant; it’s a suite of solutions integrated across the Microsoft Cloud. It’s designed to boost your productivity by providing enterprise-grade data security and privacy features. Whether you’re coding, creating content, or analyzing data, Copilot is there to streamline your workflow.
Getting Started with Copilot To get started, dive into the wealth of resources available on the official Copilot page. From curated training and documentation to informative videos and playlists, there’s a treasure trove of knowledge waiting for you.
Customizing Your Experience One of the most exciting aspects of Copilot is its flexibility. You can expand and enrich your Copilot experience with plugins, connectors, or message extensions. Even better, you can build a custom AI copilot using Microsoft Cloud technologies to create a personalized conversational AI experience.
Empowering Your Education Copilot isn’t just a tool; it’s a partner in your educational journey. It can assist you in implementing cloud infrastructure, solving technical business problems, and maximizing the value of data assets through visualization and reporting tools.
The Copilot Challenge Ready to put your skills to the test? Immerse yourself in cutting-edge AI technology and earn a badge by completing one of the unique, AI-focused challenges available until June 21, 2024. These challenges offer interactive events, expert-led sessions, and training assets to help you succeed.
Conclusion Microsoft Copilot is more than just an assistant; it’s a catalyst for innovation and productivity. As a technical student, embracing Copilot can help you stay ahead of the curve and unlock a new era of growth. So, what are you waiting for?
Let Copilot guide you through the exciting world of AI and cloud technologies. Learn how to use Microsoft Copilot | Microsoft Learn
by Contributed | May 31, 2024 | Technology
This article is contributed. See the original author and article here.
Whenever you run load tests on your endpoints in Azure Load Testing, there might be a situation where your endpoints require an access token. We have seen that before load testing these endpoints, most users generate an access token and store it in the Azure Key Vault. They then pass this token as a variable in the Header of the request as a bearer token to authenticate at the endpoint. But there might be a few problems with this approach:
- Sometime has elapsed between the act of storing the token in the key vault and the start of execution of the test. There can be a case where the token might expire during the test run and the request at the endpoints start throwing errors.
- Only one static token is stored in the key vault and is used for all the requests. When we are load testing with multiple Virtual Users, we want a situation where all users have a unique access token while making the requests, as it will mimic a real-life scenario. Currently scaling is there, but the authenticity is lost.
- Most access tokens have a lifetime of around 60-75 minutes on an average. When we are running load tests for longer durations (E.g.- Soak Tests), it is almost certain that the access token will get expired during the execution of the test. We would like to have some sort of control where the service intelligently identifies when a token is going to get expired, and renews it, so that the test can carry on as usual.
This blog covers how you can create a JMeter Test plan, where you can implement a mechanism to generate and refresh access tokens which are unique for every user and run it at scale in Azure Load Testing.
PREREQUISITES
CREATING THE TEST PLAN IN THE JMETER GUI
Create a Test Plan in JMeter. Add your variables in the “User defined variable section.” If you are testing an Azure endpoint, you may need to use the GetSecret function to store the client_ID, client_secret, tenant_ID etc. Proceed to the config elements and add a HTTP Cookie Manager and HTTP Cache Manager. Make sure to select the option to clear cache and cookies on each iteration.
Create a Thread Group and provide the required inputs. Enable the “Start Next Thread Loop” Option. Set the Loop Count to infinite and set the test duration.
Inside the Thread Group, create a Loop Controller and set the loop count to infinite. Further, inside the loop controller, add two more controllers: The transaction controller, which will generate the token and the runtime controller, which will refresh the token.
Inside the Transaction Controller, add an HTTP Sampler and setup a POST request to the token endpoint to fetch the access token. Set “grant type” as “client credentials” and “ignore Cache” as “true”. Add a HTTP Header Manager to the sampler and add the Content-Type as “application/x-www-form-urlencoded” and set the Authorization depending upon the configuration done at the token endpoint.
Add a JSON extractor to extract the access token and set it in a variable which will be used going forward.
Proceed to the runtime controller. In the Runtime section, specify the time after which you want the token to be refreshed. Usually token lasts for 60 minutes. Make sure to include some buffer time before token expiry. You can also extract the expiry time of the token, store it in a variable and pass it here.
Add another HTTP Sampler, this time sending a GET request to the target endpoint. In the HTTP Header Manager, set Authorization to Bearer ${token}.
Add some listeners like View Results tree, Summary Report etc to analyse the results locally. Results should look something like this.
Now the JMeter script is complete and can be used in Azure Load Testing to run it at scale.
Creating and Running the Load Test
- Upload the JMeter Script in the Test plan tab.
- Configure the Secrets section under the Parameters tab. Now, we are all set to launch the test.
CONCLUSION
Now you can run your load tests for an even longer duration without having to worry about the expiry of access tokens.
Try this this out and let us know if this works for you. Please use the comments section or use our feedback forum.
HAPPY LOAD TESTING!!
by Contributed | May 30, 2024 | Technology
This article is contributed. See the original author and article here.
Introduction
As a startup or an new customer exploring Microsoft Azure, you may find the terminology around identity and access management a bit perplexing. Terms like Tenant, Subscription, and Microsoft Entra ID, are crucial to understanding how to effectively manage and secure your Azure environment. This blog post aims to demystify these concepts and provide a clear, concise understanding of how they interrelate.
What is Microsoft Entra ID (Former Azure Active Directory)?
Microsoft Entra ID is the new name for Azure Active Directory (AAD), Microsoft’s cloud-based identity and access management service. It plays a crucial role in managing user, group, and application access to Azure services. Here’s a breakdown:
- Identity Management: Entra ID handles both authentication (verifying identity) and authorization (granting access) for Azure resources.
- Cloud-Based: Unlike traditional Active Directory, which is designed for on-premises environments, Entra ID operates entirely in the cloud, making it ideal for modern, cloud-first organizations.
Key Features:
- User and Group Management: You can create users and groups within your Entra ID tenant. These identities can be synchronized with your existing on-premises Active Directory using Entra Connect.
- Single Sign-On (SSO): Provides a seamless sign-on experience across multiple applications and services.
What is an Azure Tenant?
An Azure Tenant represents a dedicated instance of Microsoft Entra ID for your organization. It is essentially your organization’s space in the Azure cloud where you manage your identities and access.
Key Points:
- Created Automatically: When you sign up for Azure or any Microsoft cloud service, a tenant is automatically created.
- Domain Representation: By default, your tenant will have a domain like
yourcompany.onmicrosoft.com, which can be customized to use your own domain. (You can also add and replace your on.microsoft.com fallback domain)
What is an Azure Subscription?
An Azure Subscription is a logical container used to provision and manage Azure resources. It is closely tied to billing and acts as a boundary for resource management and deployment.
Key Characteristics:
- Resource Management: All Azure resources (e.g., virtual machines, databases) are associated with a subscription.
- Billing: Each subscription has its own billing cycle and payment terms, allowing you to manage costs effectively.
- Scalability: Best practice is to have separate subscriptions for production and non-production environments to manage permissions and scalability efficiently.
Relation to Tenant:
How They Work Together
Automatic Creation of Emtra ID Tenant:
- When you create an Azure subscription, an Entra ID tenant is automatically provisioned. This tenant manages identities and access within your subscription.
Authentication and Authorization:
- Azure subscriptions rely on Entra ID to authenticate and authorize users, services, and devices, ensuring that only authenticated identities can access Azure resources.
Role-Based Access Control (RBAC):
Multi-Subscription Management:
- A single Entra ID tenant can manage multiple subscriptions, enabling the same set of users and groups to access different environments. However, each subscription is associated with only one tenant.
Practical Tips for Startups
Benefit from the Azure setup Guide:
- Before you start building and deploying solutions by using Azure services, you need to prepare your environment. In this guide, we introduce features that help you organize resources, control costs, and secure and manage your organization. You can access it directly from the Azure Portal too.
Start with a Clear Structure:
- Separate Subscriptions: Use separate subscriptions for production and non-production environments to ensure better management and segregation of resources.
- Define Roles: Clearly define and assign Azure Roles and Entra ID Roles to manage permissions effectively.
Synchronize Identities:
- If you have an existing on-premises AD, use Entra Connect to synchronize identities with Entra ID, ensuring seamless access management.
Plan for Scalability:
- Design your subscription architecture to accommodate growth. Consider factors like billing, resource limits, and administrative boundaries.
Use Enterprise Scale Landing Zones:
- Implement recommended modular design to ensure your Azure environment can scale efficiently with your business needs.
Follow the links below if you are familiar with AWS and would like to understand how to map AWS concepts to Azure:
Conclusion
Understanding Microsoft Entra ID, Azure Subscriptions, and Tenants is essential for effective Azure governance. These concepts form the backbone of identity and access management in Azure, ensuring secure, scalable, and efficient management of your cloud resources. As your startup grows, mastering these basics will help you optimize your Azure environment, supporting your journey towards innovation and success.
For more detailed information, always refer to the latest Microsoft Azure documentation and best practices.
Recent Comments