Self-Chained APIM request limitation in internal Virtual network mode (Developer and Premium tier)

by Contributed | Nov 30, 2020 | Technology

This article is contributed. See the original author and article here.

Known Issue

When you are running API Management instance in “internal” Virtual network mode and trying to call APIs hosted in the same APIM service (use APIM gateway endpoint Url as backend Url), you may experience 500 errors with “BackendConnectionFailure”.

Below screenshots demonstrate the steps to reproduce this issue.

1. Define Recursion API/operation configuration in APIM instance. Here we let APIM forward http request https://proxy.momorin.com/recursion/echo/resource to https://proxy.momorin.com/echo/resource, which is API in the same APIM instance:

2. Try to send request through Postman, we need allow inspector trace to better troubleshoot. As a result, 500 error returned after 21 seconds.

3. Now check the APIM inspector trace for more detailed information. We can get trace url from response header “Ocp-Apim-Trace-Location”.

4. As you can see in the inspector trace, the error happens at backend level, when trying to forward request to APIM itself. Error message is “Unable to connect to the remote server”.

You checked all the network configurations, there are no NSG or force tunneling blocking the traffic from internal VNet to APIM gateway endpoint, or even though you logged into one VM inside the same VNet, the connection from this VM to APIM gateway endpoint still works well.

It’s very confusing, because you may just send a call without any other policy or any other configurations related to the backend Url and it should not fail, as the first request layer with same domain succeeded.

Internal Load Balancer Limitation

The root cause of this issue is the load balancer limitation when accessing the internal Load Balancer frontend from the participating Load Balancer backend pool VM, as documented here:

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-troubleshoot#cause-4-accessing-the-internal-load-balancer-frontend-from-the-participating-load-balancer-backend-pool-vm.

When deploying APIM service into internal VNet mode, the load balancer of gateway (proxy) endpoint is in the same subnet where APIM backend instances are deployed in.

If an internal Load Balancer is configured inside a VNet, and one of the participant backend VMs is trying to access the internal Load Balancer frontend, failures can occur when the flow is mapped to the originating VM (same VM). This scenario is not supported.

If you use APIM Premium tier, you will have at least two VMs in the subnet, so this issue may intermittently happen (traffic from instance 1 to instance2 will succeed, traffic from instance 1 back to instance 1 will fail). But if you are using Developer pricing tier, you only have one VM instance in the APIM backend pool, this issue will consistently occur.

Historically for Internal VNET mode, APIM used to override environment level DNS on the APIM VMs for Gateway hostnames (default and custom ones) to map them to loopback interface (127.0.0.1) using host file entries on the VMs so that every time Gateway (or any other software on the VM) tried to call one of these hostnames, it would connect to itself through loopback network interface defined in the host file.

After an update in February 2020, a decision was made to stop doing host file DNS overrides. This change caused outgoing traffic from APIM VM to its own hostname to be routed to APIM Load Balancer instead of loopback interface. As a result, API calls that were sent to the same APIM service via forward-request or send-request policies started failing.

Resolution

The best solution for this issue is to change the Url of the API in the policy to https://127.0.0.1 *and* add a “host” header to the request for the desired proxy host.

APIM proxy can send requests to backend (including itself) using forward-request or send-request policies. Below are the solutions for each kind of policy.

Change Url of forward-request policy

If the failing request is being sent via forward-request policy (the backendUrl of the API has been set as the Url of the APIM Proxy), the hostname of backendUrl should be changed to https://127.0.0.1. Additionally, a set-header policy should be added in <inbound> section to add the desired host header (which previously used to be part of the Url):

<policies>
    <inbound>
        <set-header name="Host" exists-action="override">
            <value>proxy.momorin.com</value>
        </set-header>
    </inbound>
    <backend>
        <forward-request />
    </backend>
</policies>

Below is one simple instruction & test result of this resolution:

200OK returned from https://proxy.momorin.com/recursion/echo/resource this time.

The forwarding url is https://127.0.0.1/echo/resource with host “proxy.momorin.com”.

Or we can apply the following in the global scope (All APIs) so that we don’t need to modify each API in the backend Url.

    <inbound>
        <choose>
            <when condition="@(context.Request.Url.ToString().Contains("proxy.momorin.com"))">
                <set-header name="Host" exists-action="override">
                    <value>proxy.momorin.com</value>
                </set-header>
                <set-backend-service base-url="@(context.Request.OriginalUrl.Scheme.ToString() + "://127.0.0.1")" />
            </when>
        </choose>
    </inbound>

Change Url of send-request policy

If APIM is called using send-request policy, host can be added directly inside the policy:

<send-request>
    <set-url>https://127.0.0.1/echo/resource</set-url>
    <set-header name="Host">
        <value>proxy.momorin.com</value>
    </set-header>
</send-request>

How to enable MSI (Managed Service Identity) for Batch compute nodes in User Subscription Mode

by Contributed | Nov 30, 2020 | Technology

This article is contributed. See the original author and article here.

Background:

Currently we can enable Azure Managed Identity to use platform-managed keys or customer-managed keys to encrypt the customer data which is stored in Azure Batch: https://docs.microsoft.com/en-us/azure/batch/batch-customer-managed-key. However, the managed identity on the Batch account is not available on the compute nodes. There was an active feature request submitted to Azure Batch team and Azure Active Directory team asking for supporting MSI in Azure Batch environment: https://feedback.azure.com/forums/269742-batch/suggestions/33640984-support-managed-service-identity#:~:text=Actually%2C%20Azure%20Batch%20is%20not,variables%20and%20clear%20text%20configuration. The implementation of this feature has begun but there is no ETA at this time. As an optional plan,  we could enable MSI for compute nodes in user subscription mode which means that user would need to manage his own Virtual Machine Scale Sets (VMSS) and those nodes are in MSI enabled environment.

Purpose:

In user subscription mode, customer can enable MSI for compute nodes directly by their own. Please note the following limitations:

1. Right now this optional plan is only valid when the pool allocation mode is user subscription which means all the compute nodes are going to be provisioned in your subscription. Please check this document for the details: https://docs.microsoft.com/en-us/azure/batch/batch-account-create-portal#create-a-batch-account


2. When these Batch VMs are provisioned every time,  for example, Batch Service creates a new Virtual Machine Scale Sets due to some scale out activity, you are required to enable the MSI manually. You can do it via Portal, PowerShell or REST API .

Pre-requirement:

• Prepare an Azure Batch account with User Subscription mode

Steps:

1. Create a new pool in the Batch account, the VMSS will be added to your subscription in a different resource group.

2. Access to the resource group and select VMSS.

3. Access to the VMSS and select the Identity tab to enable the MSI. This document provides more information about enabling system-assigned managed identify and user-assigned managed identity: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm#enable-system-assigned-managed-identity-on-an-existing-vm.

4. You will be able to modify the role assignments. In my example, I assigned Owner role of subscription in the Azure role assignments.

5. RDP to the VMSS to test if the MSI works or not. Now I can get the token to list the information of my resource group. This document lists the PowerShell command that I used in this example: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/tutorial-windows-vm-access-arm#get-an-access-token-using-the-vms-system-assigned-managed-identity-and-use-it-to-call-azure-resource-manager.

$response = Invoke-WebRequest -Uri ‘http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/‘ -Method GET -Headers @{Metadata=”true”}

$content = $response.Content | ConvertFrom-Json

$ArmToken = $content.access_token

(Invoke-WebRequest -Uri ‘https://management.azure.com/subscriptions/a2d49d28-b5b1-48fe-83dc-ada50a035a99/resourceGroups/moshi?api-version=2016-06-01‘ -Method GET -ContentType “application/json” -Headers @{ Authorization=”Bearer $ArmToken”}).content

Lesson Learned #148: High wait time due to Parameter Sniffing

by Contributed | Nov 30, 2020 | Technology

This article is contributed. See the original author and article here.

This last week I had a service request where our customer faced a high wait time due to a parameter sniffing. In this situation, our customer is not able to change any option of the query and they asked about if there is any other option to fix this issue.

We have several alternatives but I would like to share 3 of them besides other ones:

Alternative 1

• I developed an application that is running a parametrized query like our customer has.

• As you could see we have a process that creates a cached plan.

SELECT plan_handle,UseCounts,RefCounts, Cacheobjtype, Objtype,  TEXT AS SQL 
FROM sys.dm_exec_cached_plans 
CROSS APPLY sys.dm_exec_sql_text(plan_handle) 
where text like '%SELECT count(Id) FROM PerformanceVarcharNVarchar%' AND objtype='Prepared'

• Once, you have identified the plan_handle, you could run the following DBCC FREEPROCCACHE(plan_handle) , you are going to clean the cache plan for this query and you are going to have a new execution plan with this compile value. Information: DBCC FREEPROCCACHE (Transact-SQL) – SQL Server | Microsoft Docs

Alternative 2

• Other option that you have is to disable parameter sniffing in the database properties. But, you could have other performance issue for the rest of the queries.

Alternative 3

• 
  

  Finally, you could use plan guide, for example:

  
  

   

  
  
  
  
  • My C# is converted the TSQL is this way: (@Name nvarchar(200))SELECT count(Id) FROM PerformanceVarcharNVarchar Where TextToSearch = @Name
  
  
  
  
  • If you need this info you could see it using the following TSQL if needed:
  
  
  
  
  
  

SELECT TOP 2500
       databases.name,
       dm_exec_sql_text.text AS TSQL_Text,
       CAST(CAST(dm_exec_query_stats.total_worker_time AS DECIMAL)/CAST(dm_exec_query_stats.execution_count AS DECIMAL) AS INT) as cpu_per_execution,
       CAST(CAST(dm_exec_query_stats.total_logical_reads AS DECIMAL)/CAST(dm_exec_query_stats.execution_count AS DECIMAL) AS INT) as logical_reads_per_execution,
       CAST(CAST(dm_exec_query_stats.total_elapsed_time AS DECIMAL)/CAST(dm_exec_query_stats.execution_count AS DECIMAL) AS INT) as elapsed_time_per_execution,
       dm_exec_query_stats.creation_time, 
       dm_exec_query_stats.execution_count,
       dm_exec_query_stats.total_worker_time AS total_cpu_time,
       dm_exec_query_stats.max_worker_time AS max_cpu_time, 
       dm_exec_query_stats.total_elapsed_time, 
       dm_exec_query_stats.max_elapsed_time, 
       dm_exec_query_stats.total_logical_reads, 
       dm_exec_query_stats.max_logical_reads,
       dm_exec_query_stats.total_physical_reads, 
       dm_exec_query_stats.max_physical_reads,
       dm_exec_query_plan.query_plan,
       dm_exec_cached_plans.cacheobjtype,
       dm_exec_cached_plans.objtype,
       dm_exec_cached_plans.size_in_bytes,*
FROM sys.dm_exec_query_stats 
CROSS APPLY sys.dm_exec_sql_text(dm_exec_query_stats.plan_handle)
CROSS APPLY sys.dm_exec_query_plan(dm_exec_query_stats.plan_handle)
INNER JOIN sys.databases
ON dm_exec_sql_text.dbid = databases.database_id
INNER JOIN sys.dm_exec_cached_plans 
ON dm_exec_cached_plans.plan_handle = dm_exec_query_stats.plan_handle
WHERE NAME='DotNetExample'
and dm_exec_sql_text.text like '%SELECT count(Id) FROM PerformanceVarcharNVarchar%'
ORDER BY tsql_text DESC;

• I created a plan guide recompiling the query every time that I executed the query in this way:

EXEC sp_create_plan_guide   
    @name =  N'Guide1',  
    @stmt = N'SELECT count(Id) FROM PerformanceVarcharNVarchar Where TextToSearch = @Name',  
    @type = N'SQL',  
    @module_or_batch = null,  
    @params = N'@Name nvarchar(200)',  
    @hints = N'OPTION (RECOMPILE)';

• Finally, if I need to drop the plan guide, basically, I could run the following command to delete it:  

EXEC sp_control_plan_guide N'DROP', N'Guide1';  

Enjoy!

MSIX – Batch Conversion of your App-V 5 Packages

by Contributed | Nov 29, 2020 | Technology

This article is contributed. See the original author and article here.

Hello everyone, this is Ingmar Oosterhoff, a Customer Engineer at Microsoft. In an earlier post I described how to set up an environment for a bulk conversion of  .msi and .exe installers. But what about App-V? In this blog I will explain how easy it is to batch convert your App-V 5 packages to MSIX.

Create new MSIX package

The MSIX packaging tool natively converts a single App-V package to MSIX, removing the requirement to re-package them and a feature we will make use of.

Let us start by preparing the host machine. In this case, my laptop. Components needed are contained in the shopping list below.

The shopping list: 

• MSIX Packaging tool 


• Folder containing App-V packages. 


• A Signing certificate. 


• Signtool.exe 


• Conversion script


• Xml template

The MSIX Packaging Tool is free and can be installed from the Microsoft Store, so that is an easy step.

 For the folder containing App-V packages, I am going to use my App-V Content Share. However, any folder that contains App-V packages will suffice.

Please keep in mind that User- and/or DeploymentConfig files of the App-V packages will be ignored.

Signing Certificate 

I will need to sign the newly created MSIX packages with a certificate, so on my laptop on my C: drive I have created a folder named MSIX. In this folder I have created a folder Signing, this folder contains the certificate, used to sign the packages, and signtool.exe, which is part of the Windows 10 SDK (Software Developer Kit). (Have a look at our earlier post on how to set that up)

The Script and xml template

In the MSIX folder I created earlier on the C: drive, I have a folder BatchConversion  from my last post, and created an additional PowerShell file named batch_convert_appv.ps1, and the template file MsixPackagingToolTemplate.xml

batch_convert_appv.ps1

The script below was created together with my fellow Customer Engineer Ryan Cobb. This script iterates through the list of App-V packages and converts them to MSIX. All that is needed is to copy the script and xml template from below, modify the following parameters within the script. 

$AppvContentStore = "C:repositoryApp-VPackages"
$PublisherName = "CN=Contoso Software (FOR LAB USE ONLY), O=Contoso Corporation, C=US"
$PublisherDisplayName = "Contoso"
$counter = 1
$Certificate = "C:MSIXSigningContosoLab.pfx"
$CertificatePassword = "notreallythecertificatepassword"
# Creating a folder to store the template files used for the conversion
New-Item -Force -Type Directory ([System.IO.Path]::Combine($workingDirectory, "MPT_Templates"))
# Creating a folder to store the MSIX packages
New-Item -Force -Type Directory ([System.IO.Path]::Combine($workingDirectory, "MSIX"))
# get all the App-V packages from the ContentStore
get-childitem $AppvContentStore -recurse | Where-Object { $_.extension -eq ".appv" } | ForEach-Object {
    $Installerpath = $_.FullName
    $filename = $_.BaseName
    write-host "starting the conversion of: " $Installerpath
    # MSIX package name cannot contain spaces, dashes or dots, so replacing these
    $packageStrippedName = $filename -replace 's+', '' -replace '.', '' -replace '-', ''
    $job = "job" + $counter
    
    # get the contents of the template XML
    [String]$newXml = Get-Content -path $PSScriptRootMsixPackagingToolTemplate.xml | Out-String
    # Replace the placeholders with the correct values
    $newXml = $newXml.Replace("[Installer]", "$Installerpath")
    $newXml = $newXml.Replace("[SaveLocation]", "$SaveLocation")
    $newXml = $newXml.Replace("[PackageName]", "$packageStrippedName")
    $newXml = $newXml.Replace("[PackageDisplayName]", "$filename")
    $newXml = $newXml.Replace("[PublisherName]", "$PublisherName")
    $newXml = $newXml.Replace("[PublisherDisplayName]", "$PublisherDisplayName")
    # saving the newly created template
    $newXml | out-File $MPTtemplateMsixPackagingToolTemplate_$job.xml -Encoding Ascii -Force
    # Starting the conversion
    MsixPackagingTool.exe create-package --template "$MPTtemplateMsixPackagingToolTemplate_$job.xml"
    MsixPackagingTool.exe cleanup
    $counter = $counter + 1
}
# App-V packages converted to MSIX. Signing the new MSIX packages
Get-ChildItem $msixFolder | foreach-object {
    $MSIXpackage = $_.FullName
    C:MSIXSigningsigntool.exe sign /a /v /fd SHA256 /f $Certificate /p $CertificatePassword "$MSIXpackage"

MsixPackagingToolTemplate.xml

Below the contents of the xml template file

<MsixPackagingToolTemplate
    xmlns="http://schemas.microsoft.com/appx/msixpackagingtool/template/2018"
    xmlns:mptv2="http://schemas.microsoft.com/msix/msixpackagingtool/template/1904">
<Installer Path="[Installer]"/>
<SaveLocation PackagePath="[SaveLocation]" />
<PackageInformation
    PackageName="[PackageName]"
    PackageDisplayName="[PackageDisplayName]"
    PublisherName="[PublisherName]"
    PublisherDisplayName="[PublisherDisplayName]"
    Version="1.0.0.0">
</PackageInformation>
</MsixPackagingToolTemplate>

Once all the changes have been made and the script saved Batch Conversion can begin.

Open a PowerShell window as an administrator and change location to the Batch Conversion folder where the script is stored.

Type .batch_convert_appv.ps1 and press enter

The script will convert all the App-V packages to signed MSIX packages and store them in a subfolder named MSIX.

Happy converting! Let me know how it went!

Ingmar Oosterhoff, Ryan Cobb, and Matthias Herfurth

Disclaimer
The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.