This article is contributed. See the original author and article here.

Overview


This blog demonstrates a multi-tier application deployment on to Azure Kubernetes Service along with several other Azure managed services such as Azure Database for MySQL, Azure Functions, etc.


Note: There may be few features that are used in this blog such as Azure Active Directory Pod Identity are still in preview, these features are not recommended for production deployment. 


 


Architecture


Architecture.jpg


 


Setup


We will create and setup the infrastructure including the following services:



  1. Azure Container Registry for storing images

  2. AAD Enabled, Managed AKS Cluster with the below addons and components

    1. Application Gateway Ingress Controller Addon

    2. Monitoring Addon

    3. LetsEncrypt for Certificate authority

    4. KEDA runtime for Azure Functions on Kubernetes clusters



  3. Azure Database for MySQL Service

  4. Azure Storage Queues

  5. DNS Zone for custom domain

  6. SendGrid Account for email service


 


Cluster Creation


Clone repository


git clone https://github.com/ssarwa/multitiered-app-on-azure
cd multitiered-app-on-azure
# You could use script.azcli as your working file. Don’t run the script as is!

 


Initialize variables


# Add variables (sample values below change as required)
resourcegroupName=’CNCF-Azure-RG’
clusterName=’myaksCluster’
location=’westus’
appGtwyName=’AKSAppGtwy’
acrName=’cncfazure’
domainName=’sarwascloud.com’
dnsRG=’dns-rg’
subDomain=’expense’
mysqlSvr=’expensedbserver’
adminUser=’expenseadmin’
mysqlPwd=”
keyvaultName=’expensesvault’
# Identity name must be lower case
identityName=’exppoidentity’
# Storage name must be lower case and globally unique
storageAcc=’expensesqueue’
subscriptionId=’12bb4e89-4f7a-41e0-a38f-b22f079448b4′
tenantId=’72f988bf-86f1-41af-91ab-3d7cd011db47′

 


Login to Azure


az login
az account set -s $subscriptionId

 


Register to AKS preview features


# Follow https://docs.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity
az feature register –name EnablePodIdentityPreview –namespace Microsoft.ContainerService
az provider register -n Microsoft.ContainerService
az extension add –name aks-preview
az extension update –name aks-preview

 


Create Resource Group


az group create –name $resourcegroupName –location $location

 


Create ACR


az acr create –resource-group $resourcegroupName –name $acrName –sku Standard

 


Get Object ID of the AAD Group (create AAD Group and add the members, in this case: expenses-ad-azure)


This group is needed as admin group for the cluster to grant cluster admin permissions. You can use an existing Azure AD group, or create a new one. Record the object ID of your Azure AD group.


az ad group create –display-name expenses-ad-azure –mail-nickname expenses-ad-azure –description ‘Group for Managing AAD based AKS cluster’
objectId=$(az ad group list –filter “displayname eq ‘expenses-ad-azure'” –query ‘[].objectId’ -o tsv)

 


Create an AKS-managed Azure AD cluster with AGIC add-on and AAD Pod Identity


az aks create
-n $clusterName
-g $resourcegroupName
–network-plugin azure
–enable-managed-identity
-a ingress-appgw –appgw-name $appGtwyName
–appgw-subnet-cidr “10.2.0.0/16”
–enable-aad
–enable-pod-identity
–aad-admin-group-object-ids $objectId
–generate-ssh-keys
–attach-acr $acrName

# Enable monitoring on the cluster
az aks enable-addons -a monitoring -n $clusterName -g $resourcegroupName


 


Add Public IP to custom domain


# Get Node Resource Group
nodeRG=$(az aks show –resource-group $resourcegroupName –name $clusterName –query nodeResourceGroup -o tsv)

# Get Public IP created by App Gtwy in AKS created cluster
appIP=$(az network public-ip show -g $nodeRG -n $appGtwyName-appgwpip –query ipAddress -o tsv)

# Create DNS zone, if not created
az network dns zone create -g $dnsRG -n $domainName

# Once created, add Nameservers in the domain provider (eg go daddy, may take sometime to update the name servers)
az network dns record-set a add-record –resource-group $resourcegroupName –zone-name $domainName –record-set-name $subDomain –ipv4-address $appIP


For more details see the tutorial on registering custom domain: https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns 


 


Connect to the Cluster


Merge Kubeconfig


az aks get-credentials –resource-group $resourcegroupName –name $clusterName –admin

 


Install Cert Manager


# Install the CustomResourceDefinition resources separately
# Note: –validate=false is required per https://github.com/jetstack/cert-manager/issues/2208#issuecomment-541311021
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.13/deploy/manifests/00-crds.yaml –validate=false

kubectl create namespace cert-manager
kubectl label namespace cert-manager cert-manager.io/disable-validation=true
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager –namespace cert-manager –version v0.13.0 jetstack/cert-manager
kubectl apply -f yml/clusterissuer.yaml

# Test a sample application. The below command will deploy a Pod, Service and Ingress resource. Application Gateway will be configured with the associated rules.
sed -i “s//$domainName/g” yml/Test-App-Ingress.yaml
sed -i “s//$subDomain/g” yml/Test-App-Ingress.yaml
kubectl apply -f yml/Test-App-Ingress.yaml

# Clean up after successfully verifying AGIC
kubectl delete -f yml/Test-App-Ingress.yaml


 


Install KEDA runtime


helm repo add kedacore https://kedacore.github.io/charts
helm repo update
kubectl create namespace keda
helm install keda kedacore/keda –namespace keda

 


Install CSI Provider for Azure KeyVault


helm repo add csi-secrets-store-provider-azure https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
helm repo update
kubectl create namespace csi
helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure –namespace csi

 


Assign managed identity


clientId=$(az aks show -n $clusterName -g $resourcegroupName –query identityProfile.kubeletidentity.clientId -o tsv)
scope=$(az group show -g $nodeRG –query id -o tsv)
az role assignment create –role “Managed Identity Operator” –assignee $clientId –scope $scope

 


Create Azure KeyVault for saving secrets and assign identity


az keyvault create –location $location –name $keyvaultName –resource-group $resourcegroupName
kvscope=$(az keyvault show -g $resourcegroupName -n $keyvaultName –query id -o tsv)
az identity create -g $resourcegroupName -n $identityName
idClientid=$(az identity show -n $identityName -g $resourcegroupName –query clientId -o tsv)
idPrincipalid=$(az identity show -n $identityName -g $resourcegroupName –query principalId -o tsv)
identityId=$(az identity show -n $identityName -g $resourcegroupName –query id -o tsv)
az role assignment create –role “Reader” –assignee $idPrincipalid –scope $kvscope

# Set permissions
az keyvault set-policy -n $keyvaultName –secret-permissions get –spn $idClientid

# Add Pod Identity
az aks pod-identity add –resource-group $resourcegroupName –cluster-name $clusterName –namespace default –name $identityName –identity-resource-id $identityId

 


Create MySQL managed service (basic sku) and add Kubernetes Load Balancer’s public ip in it firewall rules


aksPublicIpName=$(az network lb show -n kubernetes -g $nodeRG –query “frontendIpConfigurations[0].name” -o tsv)
aksPublicIpAddress=$(az network public-ip show -n $aksPublicIpName -g $nodeRG –query ipAddress -o tsv)
az mysql server create –resource-group $resourcegroupName –name $mysqlSvr –location $location –admin-user $adminUser –admin-password $mysqlPwd –sku-name B_Gen5_2
az mysql server firewall-rule create –name allowip –resource-group $resourcegroupName –server-name $mysqlSvr –start-ip-address $aksPublicIpAddress –end-ip-address $aksPublicIpAddress

# Replace with your Local Machine IP. You can use: https://www.whatsmyip.org/
az mysql server firewall-rule create –name devbox –resource-group $resourcegroupName –server-name $mysqlSvr –start-ip-address <Dev station ip> –end-ip-address <Dev station ip>


 


Login to MySQL (you may need to add you ip to firewall rules as well). # Login to MySQL Client your your local dev box: sudo apt install mysql-client-core-8.0


mysql -h $mysqlSvr.mysql.database.azure.com -u $adminUser@$mysqlSvr -p
show databases;

CREATE DATABASE conexpweb;

CREATE DATABASE conexpapi;
USE conexpapi;

CREATE TABLE CostCenters(
CostCenterId int(11) NOT NULL,
SubmitterEmail text NOT NULL,
ApproverEmail text NOT NULL,
CostCenterName text NOT NULL,
PRIMARY KEY ( CostCenterId )
);

# Insert example records
INSERT INTO CostCenters (CostCenterId, SubmitterEmail,ApproverEmail,CostCenterName) values (1, ‘ssarwa@microsoft.com’, ‘ssarwa@microsoft.com’,’123E42′);
INSERT INTO CostCenters (CostCenterId, SubmitterEmail,ApproverEmail,CostCenterName) values (2, ‘ssarwa@microsoft.com’, ‘ssarwa@microsoft.com’,’456C14′);
INSERT INTO CostCenters (CostCenterId, SubmitterEmail,ApproverEmail,CostCenterName) values (3, ‘ssarwa@microsoft.com’, ‘ssarwa@microsoft.com’,’456C14′);

# Verify Records
SELECT * FROM CostCenters;

quit


 


Create Storage queue


az storage account create -n $storageAcc -g $resourcegroupName -l $location –sku Standard_LRS
# Do not change queue name of contosoexpenses. KedaFunction queue trigger relies on this queue name.
az storage queue create -n contosoexpenses –account-name $storageAcc

 


Add corresponding secrets to the create KeyVault



  1. MySQL Connection strings (choose ADO.NET) – both for API and Web

    1. mysqlconnapi

    2. mysqlconnweb



  2. Storage Connection strings

    1. storageconn



  3. Sendgrid Key

    1. sendgridapi




az keyvault secret set –vault-name $KeyVault –name mysqlconnapi –value ‘<replace>Connection strings for MySQL API connection</replace>’
az keyvault secret set –vault-name $keyvaultName –name mysqlconnweb –value ‘<replace>Connection strings for MySQL Web connection</replace>’
az keyvault secret set –vault-name $keyvaultName –name storageconn –value ‘<replace>Connection strings for Storage account</replace>’
# Make sure to register for free SendGrid Account and verify identity. Visit https://sendgrid.com
az keyvault secret set –vault-name $keyvaultName –name sendgridapi –value ‘<replace>Sendgrid Key</replace>’
az keyvault secret set –vault-name $keyvaultName –name funcruntime –value ‘dotnet’

 


Application Deployment



registryHost=$(az acr show -n $acrName –query loginServer -o tsv)

az acr login -n $acrName

cd source/Contoso.Expenses.API
docker build -t $registryHost/conexp/api:latest .
docker push $registryHost/conexp/api:latest

cd ..
docker build -t $registryHost/conexp/web:latest -f Contoso.Expenses.Web/Dockerfile .
docker push $registryHost/conexp/web:latest

docker build -t $registryHost/conexp/emaildispatcher:latest -f Contoso.Expenses.KedaFunctions/Dockerfile .
docker push $registryHost/conexp/emaildispatcher:latest

cd ..

# Update yamls files and change identity name, keyvault name, queue name and image used refer values between <> in all files
# Create CSI Provider Class
# Use gsed for MacOS
sed -i “s/<Tenant ID>/$tenantId/g” yml/csi-sync.yaml
sed -i “s/<Cluster RG Name>/$resourcegroupName/g” yml/csi-sync.yaml
sed -i “s/<Subscription ID>/$subscriptionId/g” yml/csi-sync.yaml
sed -i “s/<Keyvault Name>/$keyvaultName/g” yml/csi-sync.yaml
kubectl apply -f yml/csi-sync.yaml

# Create API
sed -i “s/<identity name created>/$identityName/g” yml/backend.yaml
sed -i “s/<Backend image built>/$registryHost/conexp/api:latest/g” yml/backend.yaml
sed -i “s/<Keyvault Name>/$keyvaultName/g” yml/backend.yaml
kubectl apply -f yml/backend.yaml

# Create frontend
sed -i “s/<identity name created>/$identityName/g” yml/frontend.yaml
sed -i “s/<frontend image built>/$registryHost/conexp/web:latest/g” yml/frontend.yaml
sed -i “s/<Keyvault Name>/$keyvaultName/g” yml/frontend.yaml
kubectl apply -f yml/frontend.yaml

# Create ingress resource
sed -i “s/<custom domain name>/$domainName/g” yml/ingress.yaml
sed -i “s//$subDomain/g” yml/ingress.yaml
kubectl apply -f yml/ingress.yaml

# Create KEDA function
sed -i “s/<identity name created>/$identityName/g” yml/function.yaml
sed -i “s/<function image built>/$registryHost/conexp/emaildispatcher:latest/g” yml/function.yaml
kubectl apply -f yml/function.yaml


Once the ingress controller updates with new frontend deployed it may take a min for Application gateway to update.


Browse the application URL: https://subdomain.yourcustomdomain.com


 


Next Steps



  • Implement Service Mesh (like OSM) for securing service to service communications

  • Enabling managed identity to access MySQL and Storage services, thus removing Key Vault references

  • Enabling Github Actions for CI/CD pipelines


 

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.