Accessing blobs from specific directory inside container using event grid trigger

by Contributed | Jan 5, 2021 | Technology

This article is contributed. See the original author and article here.

Storage Accounts with Data lake Storage gen2 enabled allows us to create directories within container.

When we use When a blob is added or modified (properties only) trigger. It works fine for destination where we are looking for blobs inside containers but not inside any directories.

For example:

• It works for these blobs which are directly inside container.

• It won’t work for these blobs which are present inside directory in container.

Even though we choose the location of this directory for When a blob is added or modified (properties only) trigger. The trigger won’t get fired when new blob is created or modified inside directory.

To overcome this, we can use event grid trigger When a resource event occurs.

Make sure Microsoft.EventGrid is registered under resource providers in subscription.

This is webhook type trigger and it registers a webhook with the endpoint. In this case it will be a specified path inside container of a storage account.

• In Event Type we can choose from the given list.


• We need to add a new parameter named prefix filter.


• if we want to register a webhook with any container we can give:  /blobServices/default/containers/containerName inside prefix filter.


• If we want to give path of specific directory, we need to append /blobs after container name and then directory names after /blobs.

For example:

• for a directory named folder1 inside container container1 the path would be:

         /blobServices/default/containers/container1/blobs/folder1/

Note: the last forward slash ‘/’ after directory name is very important, otherwise you would not get the desired result.

Getting the content of those blobs:

For getting the content inside blobs we can use either HTTP action or get blob content (using path) action.

This is the data object returned after trigger got fired.

“data”: {

    “api”: “PutBlob”,

    “clientRequestId”: “clientRequestIdValue”,

    “requestId”: “requestIdValue”,

    “eTag”: “eTagValue”,

    “contentType”: “text/plain”,

    “contentLength”: 6,

    “blobType”: “BlockBlob”,

    “blobUrl”: “https://storageAccountName.blob.core.windows.net/containerName/folderName/fileNameWithExtension“,

    “url”: “https://storageAccountName.blob.core.windows.net/containerName/folderName/fileNameWithExtension“,

    “sequencer”: “sequencerValue”,

    “storageDiagnostics”: {

      “batchId”: “batchIdValue”

    }

  }

We can use either blobUrl or url element from data value to retrieve the path for getting blob content.

• We can use managed identity for HTTP action to get the content from URL.

Expression Inside URI :  triggerBody()?[‘data’]?[‘blobUrl’]

You can also use below expression to frame audience from url element of data object.

concat(split(triggerBody()?[‘data’]?[‘url’],’net’)[0],’net’)

• Using get blob content :

Expression inside Blob: split(triggerBody()?[‘data’]?[‘blobUrl’],’.net/’)[1]

Access Storage Accounts behind Firewalls from Logic Apps within the same region

by Contributed | Jan 5, 2021 | Technology

This article is contributed. See the original author and article here.

Storage accounts behind firewalls will not be accessible by Logic apps within the same region using Azure blob storage connector, because we can not add firewall rules to give access for the connector outgoing IP addresses; please check the following link for details: https://docs.microsoft.com/en-us/azure/connectors/connectors-create-api-azureblobstorage#access-storage-accounts-behind-firewalls

Azure blob storage connector will be returning 403 error, as shown below:

As a workaround, we use the HTTP action to call storage account REST API’s using managed identity; to do so, please follow the steps below:

1. Enable your logic app managed identity:
   
   
   
   1. Go to your logic app.
   
   
   2. Go to Settings -> Identity.
   
   
   3. Select System assigned.
   
   
   4. Select Status -> On.
   
   
   5. Click Save.                                                                                                                                   
   
   
   
   


2. In your storage account, add role assignment for your logic app:
   
   
   
   1. Go to your storage account.
   
   
   2. Click on access Control (IAM).
   
   
   3. Click on Role assignments tab.
   
   
   4. Click Add and select Add role assignment:
      
      
      
      1. From the Role dropdown list, select Storage Blob Data Contributor.
      
      
      2. From Assign access to dropdown list, select Logic App.
      
      
      3. Under the Select field, type the name of your logic app to search for it and select it once it’s found, it will appear under the selected members list.
      
      
      4. Click Save.                                                                                                                         
      
      
      
      
   
   
   
   


3. Connect your logic app to storage account using HTTP action to call storage account REST API:
   
   
   
   1. Go to your logic app
   
   
   2. Click on View in logic apps designer
   
   
   3. Add an action
   
   
   4. Search for and select HTTP action:
      
      
      
      1. From method dropdown list, select a method (GET, PUT, etc.).
      
      
      2. Enter the URI for the folder under your storage account (https://your account name.blob.core.windows.net/container name/file Name)
      
      
      3. Under headers, add the following key value pairs:
         
         
         
         1. Key: x-ms-blob-type, Value: BlockBlob.
         
         
         2. Key: x-ms-version, Value: 2019-07-07.
         
         
         
         
      
      
      4. In the body field, enter or select the variable or data token that holds your file content, in case of using PUT method.
      
      
      5. From add new parameter, select authentication:
         
         
         
         1. From Authentication type dropdown list, select Managed Identity.
         
         
         2. From Managed Identity dropdown list, select System Assigned Managed Identity.
         
         
         3. For audience, enter https://storage.azure.com                                                        
         
         
         
         
      
      
      
      
   
   
   
   

Import Project Desktop files to Project for the web API

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

You can now use an API through Powershell to import .mpp files into Project for the web! 

The instructions below will use your credentials to create a project from the .mpp file.  

Instructions: 

1.Download these files (link) and put them under one folder 
2.Open Powershell and navigate to that folder 

3* .Run this command Import-Module .ProjectImport.psm1 

4**.Run this command ImportProject –InstanceId your instance ID –FilePath “Your full filepath” * 

When Project has finished importing your .mpp file, you will see a success message with a URL to your new project. From here, your imported project will behave exactly like any other project in Project for the web. 

*See the FAQ, Tips, & Tricks section below if you have an issue with these steps! 

**To find your Instance ID: 

a. In the Dynamics 365 Administration Center, select the default instance, and then click Open. 

 b. On the PowerApps setting page, look at the first part of the URL to determine your Instance ID value. In the graphic below, the Instance ID value would be https://orgde6d15d8.crm.dynamics.com. 

FAQ, Tips, & Tricks 

Q: Can I import tasks from my .mpp file into an existing project? 

A: We do not currently support project updates through .mpp import. Importing a .mpp file will create a new project. 

Q: Are there any limits on what .mpp files I can import through the API? 

A: The normal Project for the web limits apply to imported .mpp files. If you attempt to import a .mpp file that violates these limits, you will see an error when importing. You can find these limits here.  

Q: I’m seeing an error when I try to run the first Powershell instruction. What should I do? 

A:  If you’re seeing an error, try unblocking the dll files then run this instruction again.  

To unblock a file: 

1. Right-click the file and open the Properties pane 


2. Scroll to the bottom of the pane to the Security section 


3. Toggle the ‘Unblock’ toggle in this section 


4. Click Apply 

Once you’ve unblocked both files, you should re-attempt to import your project.  

Q: Some of my data wasn’t there when I opened my imported project. What’s going on? 

A: Project for the web has does not support all features included in Project Desktop. To make it possible to import .mpp files into Project for the web, some data must be dropped to fit into the new environment.  

Unsupported features include (but are not limited to): 

• Baseline  

• Project-level custom fields  


• Cross-project dependencies  


• Links  


• Schedule from setting  


• Sub Projects  

• Constraints  


• Deadlines  


• Formulas  


• Null tasks  


• Inactive tasks  

• Manual tasks  


• Recurring tasks  


• Resources 

Please let us know what you think in the comments!

Troubleshooting Third Party Updates in MEM CM Hierarchy

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

The intention of this blog is not to go into the details of every aspect of third party patching feature in MEM CM as that is covered in a lot of blogs and our own documentation itself – Enable third party updates – Configuration Manager | Microsoft Docs. This is to run through some of the basic requirements which could help troubleshoot issues that you may run into while setting up the feature in a MEM CM hierarchy.

Although much of this is true for a single Primary, I am going to cover the scenario when you have a CAS and Primary in place as that makes it a bit more complicated. To add to it, in my case I have the SUP role running on a remote site system as that has a bearing on the set up.

High level data flow diagram

Catalog List

There are two types of Catalogs in the MEM CM third party patching, Partner Catalogs and Custom Catalogs. Partner Catalogs are currently HP, Dell and Lenovo. Custom Catalog for eg:- is Adobe. Both are catalogs published by third parties but the there is a difference in the way the communication flows for each and hence why I am calling it out.

General Pre-requisites

I know most of it is duplicate from the Microsoft docs site but I will call it out just so you have it for reference while reading this blog.

1. Enough disk space on the top-level update point’s WSUS Content folder to store the source binary for third party updates. In our case the top level is the CAS and the SUP on CAS is configured to store the content.


2. The third-party software update synchronization requires internet access.




• For partner catalog list, download.microsoft.com over HTTPS port 443 is needed. Only if this is allowed the HP, DELL and Lenovo entries will appear in the console. This connection goes out from the SUP and if the SUP can get to the above said URL, the next full sync of the SUP, which is typically every 7 days will populate the catalog entries.


• Third party updates use the same proxy settings as SUP and the proxy should allow the required URLs for the respective update catalog for the sync to work.

Additional Pre-Requisites on the SUP as the SUP is remote in this case

1. Configure SSL on WSUS/SUP (assumption is that is remote.)


2. Choose between Configuration Manager Generated Self-Signed WSUS certificate for signing third party updates and for this to work or choose your own code signing cert issued by your Organisation.




• Remote registry should be enabled on SUP server


• The WSUS server connection account should have remote registry permissions on the SUP/WSUS server.




3. HKLMSoftwareMicrosoftUpdate ServicesServerSetup, create a new DWORD named EnableSelfSignedCertificates with a value of 1.


4. If you are using the self-signed cert the cert needs to be Trusted Publisher store and Trusted root store on the CAS and the WSUS. If the WSUS server connection account has remote admin permissions on the WSUS this will be done by the system itself but make sure these certs exist. The client will also get the cert added to its trusted root with the next MEM CM client policy update. If using a PKI issued cert, make sure that authority is trusted all on relevant systems including the client.


5. The machine where the console is running is used to download the updates from WSUS and add it to the updates package if you are doing it manually. The WSUS signing certificate must be trusted on the console machine. If it isn’t, you may see issues with the signature check during the download/sync of third-party updates.


6. When the third-party software update synchronization service on the top-level software update point requires a proxy server for internet access, digital signature checks may fail. To mitigate this issue, configure the WinHTTP proxy settings on the site system. For more information, see Netsh commands for WinHTTP.

Issue 1:- Cannot see the partner catalogs list in console.

1. The synchronization of the partner catalogs is triggered by the WSUS via the SMS_ISVUPDATES_SYNCAGENT component. The component connects to download.microsoft.com using the proxy configured for your site system (in this case the WSUS). The go.microsoft.com link in the log re-directs to this page. Make sure your site system can connect to this URL. Check the SMS_ISVUPDATES_SYNCAGENT.log for similar entries as in the image above for a success.


2. At this point if you see errors along the lines of “Unable to connect to site database“in the ISV sync log on WSUS and the partner catalogs not being populated, check the permissions for the SUP machine account on the SCCM database.


3. Once all is working in terms of connectivity it takes a full sync for this to appear which is by default every 7 days.

Issue 2:- Console does not show signing certificate information in WSUS Signing Certificate Configuration.

SUP Config for CAS

In the above image I have chosen the option to use WSUS self-signing certificates. Once you choose that option the area that shows WSUS signing certificate details will be empty. If that option doesn’t populate after a few WSUS syncs check the below steps.
Note:- If you are manually managing the certificate please refer to Enable third party updates – Configuration Manager | Microsoft Docs

1. For the above information to be published and be available you need to turn on https for WSUS. Remember this is only for the CAS SUP as that is where this config gets added and synced. If you don’t turn on the https as below you would see an error in the logs.
   

Require SSL communication to WSUS server must be selected.

2. Once the option for “Configuration Manager manages the certificate”, and WSUS is enabled on the SUP if the SUP is remote, it would need to replicate the certificate around, before it shows the details on the console. Refer to Wsyncmgr.log for entries like below.

Wsyncmgr.log on CAS

3. Once generated the cert will show up in the certificates node under security on the console as below. Make sure the valid certificate is unblocked.
   

Issue 3:- What certificates do I need and where, for this solution to work.

As I called out earlier, your SUP on CAS should be running on https which by default is on port 8531. This would mean you would have a binding of a certificate with server authentication capability on the WSUS Administration site IIS.

1. On CAS Site Server




• In trusted root certificate authorities, you need the issuing authority of the code-signing cert you used as listed in Issue 2. If you are manually managing the certificate for the code-signing part, the chances are your PKI is already in the trusted root.


• Your CAS should have the authority who issued the certificate for the SSL binding as in the above image, listed in the root certificate authorities list. If you are using self-generated cert for this (which is not the best secure way of doing it), you would have to manually do this.




2. On the CAS SUP




• Make sure the code signing cert (“WSUS Publishers Self Signed” if using Configuration Manager generated certificate) is in the Trusted Publisher’s store on the SUP. If you are using Configuration Manager generated certificate the site should add the cert by itself, but if you haven’t met all the pre-requisites this could fail and might need a manual copy.




3. On Primary SUP




• The primary SUP, which is a child of the CAS SUP, you need the issuing authority for the SSL binding cert on the CAS SUP added to the Trusted Root Certificates store. As we have set the CAS SUP to require SSL for all connections, the Primary SUP will now make an SSL connection to the parent update server.




4. On Client




• Assuming your Primary SUP is not running SSL, the only certificate that you would need on the client for specifically this is the issuing authority of the code signing certificate in the trusted roots certificates store and the trusted publishers store of the client. This applies automatically if you are using config manager generated cert for this and when you enable third party patching in client settings, via the normal policy refresh.

Issue 4:- I have everything above in place but not able to download Custom Catalog eg:- Adobe.

1.   The first step in adding a Custom Catalog is providing the details to the URL and once you set up , the process downloads the cab file and adds the certificate to MEM CM for the publisher (Adobe). This connection to the internet goes out from the machine where you are running the console using the account which launched the console. So a requirement would be for that combination to be able to get to the specified URL as in below image. This is only the first time you add the catalog. Subsequent update and syncs work from the SUP when run from schedule.
   

Note: – You would be prompted to accept a certificate to complete the subscription process, as below.

2.  If you are seeing issues publishing updates after they have synced make sure the certificate for the chosen vendor is approved in the certificates node in SCCM console.
   
   Note: – Any certs that have expired should be blocked but there are instances where you might bump into one that requires one of these expired certificates.


3. After your catalog has been subscribed you need to wait for catalog synchronization and WSUS synchronization to complete, before you can select new products in the Software Update Point (SUP) properties tab.

Issue 5:- After I configured all this I notice my Primary SUP has stopped syncing with CAS SUP.

1. If your WSUS infrastructure was not configured for SSL prior to this, one of the key changes you have made is for CAS SUP to enforce SSL connection. This means the connection from Primary to CAS will be using SSL and primary needs to trust the certificate authority that has issued the certificate used for SSL binding on CAS SUP.

Administrative Tasks

Now that we have the back end set up I would like to invite your attention to a couple of key facts

1. The initial sync after all the set up is done and category has been selected within Products within Software Update Configuration, will bring only the metadata. You need to publish the ones of your choice before you can deploy.


2. With the client settings enabled and configurations required in place clients will scan against the metadata and start showing results of the scan in the console, without you publishing the update.
   
    
   
   

   
   

   
   


3. If you have SCUP already configured in your estate, needs additional considerations to make sure it does not break the SCUP process which is not covered in this blog.

Hope this helps you in understanding the config better and troubleshoot some of the issues you might see in your environment.

Collaborate with anyone in any organization with any email address!

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

Howdy folks,

We’ve heard from you that interconnected supply and distribution chains, and vendor models are bringing B2B partners directly into your business, where secure and seamless collaboration is more important than ever. We also know how painful it can be for IT managers to keep track of guest user accounts, and for end users to remember multiple usernames and passwords. We are continually improving our Azure AD External Identities solution with more support for bring-your-own-identity (BYOI) options.

Today, we are announcing another enhancement to our BYOI story with the general availability of email-based one-time passcode (email OTP) feature for collaboration.

With email OTP, org members can collaborate with anyone in the world by simply sharing a link or sending an invitation via email. Invited users prove their identity by using a verification code sent to their email account. Once authenticated, each session providing access to the shared resource lasts 24 hours. On subsequent sign ins, users receive a new one-time code via email, which they must enter to prove continued ownership of the email account and continue receiving access.

Nicole, a marketing consultant to Woodgrove Bank, accesses Woodgrove resources by verifying her email address.

Azure AD treats email OTP-based users like other B2B guests, making them subject to security policies set by your organization such as Conditional Access, Multi-Factor Authentication (MFA) and periodic access reviews.

To get started with email OTP, check out the documentation here. As always, we invite you to share any questions or feedback about the feature through the Azure forum or @AzureAD on Twitter.

Alex Simons (@Alex_A_Simons)

Corporate Vice President of Program Management

Microsoft Identity Division

Learn more about Microsoft identity:

• Related Articles: Public preview announcement of email one-time passcode
• Return to the Azure Active Directory Identity blog home
• Join the conversation on Twitter and LinkedIn
• Share product suggestions on the Azure Feedback Forum

Azure Marketplace new offers – Volume 105

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

We continue to expand the Azure Marketplace ecosystem. For this volume, 111 new offers successfully met the onboarding criteria and went live. See details of the new offers below:

Applications 1-click AI Model Deployment by Vector AI: Vector AI allows users to encode and query vectors at the edge. TensorFlow models can be deployed with a simple upload, with no hardware to manage. Select the region for the API, the size of the model, the level of traffic, and the saved model. AdsertiCredit: Aimed at non-banking financial companies, AdsertiCredit provides credit lifecycle management and helps businesses comply with regulations in Mexico. This app is available in Spanish.  AdsertiSchool: Through e-learning and administrative functions, AdsertiSchool helps higher education institutions increase operational efficiency and improve budget utilization. This app is available in Spanish. Aqtiva: Aqtiva makes clean data accessible to everyone in your organization by implementing data quality rules and visualization with a real-time dashboard. Add quality points at the beginning of your data processes and ETL processes to make sure your data is clean and useful. ASMA – Asset Discovery and Management: ASMA is cybersecurity software that discovers the MAC and IP addresses of network assets, operating systems, and running services to help you manage them. Increase your security level by ensuring network visibility. AVEVA Unified Operations Center: AVEVA Unified Operations Center enables real-time performance management for infrastructure and process industry organizations, providing closed-loop enterprise-wide visibility to optimize assets and operations. Be:Mo: Better Mobility: Be:Mo provides APIs for automotive companies and mobility service providers, offering a single-price model for all charging stations. Be:Mo is connected to 5,000 fueling stations and more than 120,000 charging points across Europe. Bugzilla – Manage Software Development on CentOS: This ready-to-launch image from Tidal Media includes Bugzilla on CentOS. Bugzilla is a bug-tracking system used to track software defects while developing and releasing products. Bust Out Fraud Detection ML Models: ElectrifAi’s library of machine learning models identifies in advance bust-out credit card fraud, detecting users who intend to rack up charges but never pay. Get more lead time to take pre-emptive action by identifying bust-out customers before they occur. CentOS 7.8 Minimal: This ready-to-launch image from Tidal Media includes a minimal version of CentOS 7.8. CentOS minimal images are optimized for automated use at scale, with a tiny package set and a reduced security vulnerability profile. CentOS 8.2 Minimal: This ready-to-launch image from Tidal Media includes a minimal version of CentOS 8.2. CentOS minimal images are optimized for automated use at scale, with a tiny package set and a reduced security vulnerability profile. Churchill Cannibalization Demand Forecaster v5.0: Churchill’s Cannibalization Demand Forecaster (CANN) predicts the indirect impact of demand on items caused by promotions, price changes, category assortment by depth and range, and the halo effect. Churchill Replenishment Demand Forecaster v5.0: Churchill’s Replenishment Demand Forecaster (RDF) automates demand forecasting, handling high volume or fractional requirements. Forecasts can be daily, weekly, monthly, or periodic demand at the distribution center or store level. Churchill Short Life Cycle Demand Forecaster v5.0: Churchill’s Short Life Cycle Demand Forecaster (SLC) provides retailers with demand intelligence to plan and react to consumer behavior at all levels of the business, from merchandising to promotions to supply chain.  compliance platform list: Available in Italian, STUDIO INFORMATICA’s compliance platform facilitates corporate compliance, anti-money laundering, and tax investigations. CONWEAVER Linksphere: CONWEAVER Linksphere is a configurable full-stack graph platform to deploy and maintain applications for graph-typical use cases related to data context analysis. DataHub: DataHub from Information DataHub is an integrated one-stop-shop solution for businesses to gain an insight into unstructured documents and structured data. DataHub puts data discovery, data classification, and data validation at the fingertips of users in a few easy steps. DataVerify: DataVerify from Information DataHub is a SaaS solution that bridges the gap between users, receipts, invoices, and third-party apps. DataVerify reduces both the effort required to process receipts and invoices and the need for physical storage. Delphix Virtualization for Azure (3TB): Delphix is a DataOps platform for application development teams. Delphix delivers secure virtual copies of production data to test environments in minutes to streamline Microsoft Azure migration projects and remove bottlenecks from DevOps workflows. Digital Motor Vehicle Insurance Certificates: Swiftant IT Solutions India’s Digital Motor Vehicle Insurance Certificates (DiMVIC) is an end-to-end insurance policy platform. Built on Microsoft Azure, DiMVIC enables issuance of tamper-proof records stored using Azure Blockchain Service. DiLeaP – Digital Solution: DiLeaP is a dedicated and customizable training and support platform for change management teams. This tool unrolls action plans for support methodologies like Prosci. This service is available in French and English. Docker Community Edition for CentOS 7.8: This ready-to-launch image from Tidal Media includes Docker Community Edition (CE) for CentOS 7.8. Docker CE is an easy-to-deploy solution for building, assembling, and shipping container applications on Microsoft Azure. Docker: Community Edition for CentOS 8.2: This ready-to-launch image from Tidal Media includes Docker Community Edition (CE) for CentOS 8.2. Docker CE is an easy-to-deploy solution for building, assembling, and shipping container applications on Microsoft Azure. Docker Engine – Enterprise on Windows Server 2016: This ready-to-launch image from Tidal Media includes Docker Enterprise on Windows Server 2016. Docker Enterprise is a standards-based container platform for the development and delivery of modern applications. Docker Engine – Enterprise on Windows Server 2019: This ready-to-launch image from Tidal Media includes Docker Enterprise on Windows Server 2019. Docker Enterprise is a standards-based container platform for the development and delivery of modern applications. Docker Registry: Highly Scalable Server on Ubuntu: This ready-to-launch image from Tidal Media includes Docker Registry on Ubuntu. Docker Registry is a stateless, scalable server-side application for building, managing, and distributing containers, images, and artifacts. Efficient Energy Management and Control: Smart-E from G&S Management and Systems allows you to monitor, control, and analyze energy consumption based on IoT devices to optimize energy use. This app is available only in Spanish. Firstlight Media Platform: Firstlight is a modular OTT video platform that provides a turnkey solution for video service providers. The cloud-native platform is SDK-ready, containerized, microservices-based, and serverless for flexibility and scale. Flatcar Container Linux Pro: With an immutable file system, this minimal Linux distribution is Kinvolk’s Azure-optimized edition of Flatcar Container Linux, ideal for production container environments requiring the highest performance, security, and support. Forum Sentry 8.11: A secure API gateway, Forum Systems’ Forum Sentry is used for rules-based security, identity, and auditing of APIs, data, and communications. This Azure-based product features policy enforcement point (PEP) for zero-trust architecture, transaction audits, and more. Frontline ATS for Defender for Endpoint: Digital Defense’s Frontline Advanced Threat Sweep (ATS), integrated with Microsoft Defender for Endpoint, proactively scans and analyzes assets for indications of a malware infection, thwarting attacks that use dwell time to evade endpoint monitoring. Genesys Cloud Integration for Microsoft Teams: Get simplified, integrated communications and easier collaboration between Genesys Cloud contact center solution and Microsoft Teams without incurring PSTN charges. The solution enables collaboration and improves productivity. Health Cloud: Integration with P1: Pentacomp Systemy Informatyczne S.A. provides a collection of health information connectors that lets you integrate e-prescriptions and e-referrals with P1, Poland’s medical information platform for citizens. This app is available only in Polish. Imagen Digital Asset Management platform: Imagen’s platform helps businesses, sports organizations, and media companies unlock the value of their ever-growing content libraries with fast, secure, and controlled access through highly customized video management. IN-D KYC: IN-D is a digital onboarding solution with electronic know-your-customer (KYC) assisted and unassisted video options. Using AI, IN-D classifies and extracts attributes from different ID documents and also performs government validation following Indian government guidelines. Intelligent Edge – StockView for retail: Neal Analytics’ StockView uses custom AI vision models powered by Microsoft Azure to help retailers reduce lost sales and improve customer experience by automatically detecting out-of-stock items on shelves. inwink: Infinite Square’s inwink is an event-intelligence SaaS platform that helps event marketers and organizers manage the complete event lifecycle and address the needs and expectations of all stakeholders. This app is available in English and French. Leni for Working Capital Management: Larsen & Toubro Infotech’s Leni is an AI-powered analytics platform for working capital management, empowering you to proactively evaluate your cash-flow requirements with real-time insights into your accounts receivable, accounts payable, and inventories. LoDDoS – DDoS Testing Platform for Red/Blue Teams: LoDDoS is an automated distributed-denial-of-service (DDoS) testing platform that can evaluate DDoS mitigation systems/services and incident-response mechanisms without exposing an organization to a real, malicious DDoS attack. Martello Gizmo – Microsoft 365 Digital Experience Monitoring: Martello Gizmo continuously tests the service quality provided by Microsoft Teams, Exchange, SharePoint, and OneDrive, providing IT teams with the metrics needed to understand problems before they affect end users. NAVOO – Digital Workplace Solution: With Arvato Systems’ NAVOO, you can centrally manage your company’s worldwide collaboration, communication, and processes. NAVOO controls the use and interaction of Office 365 functions and allows individual customization for each department. Neo4j Enterprise: Neo4j is an open-source NoSQL native graph database that provides an ACID-compliant transactional back end for your applications. The enterprise edition includes backups, clustering, and failover capabilities. OneVault – Patient Safety, Quality & Accreditation: OneVault is an integrated solution for healthcare clinical governance, risk, safety, quality, and accreditation. OneVault is easy to access and simple to navigate, providing real-time data in easily digestible formats like dashboards. Online content hosting and delivery: illuxi provides a cloud-based content hosting and delivery platform designed for organizations to host, market, and design online trainings, remote professional services, virtual events, live videos, and podcasts. OnPro Web: OnPro is a cloud-based human resources solution for HR and operations teams in the United Arab Emirates. Automate the creation and updating of your employee records and monitor changes and e-service applications as they happen. PELgx: The PELgx modular suite from BD Consultants provides various services integrated with the collection, payment, and negotiation services offered by the National Electronic Payment System (SINPE) of the Central Bank of Costa Rica (BCCR). This app is available only in Spanish. Redhat 8.2 Minimal: This ready-to-launch image from Tidal Media includes a minimal version of Red Hat 8.2. Red Hat minimal images are optimized for automated use at scale, with a tiny package set and a reduced security vulnerability profile. RSA NetWitness Platform 11.5: NetWitness Platform (RNWP) v11.5 delivers powerful new features for network detection and response, a smoother investigative workflow, expanded machine learning models, enhanced network and log capture options, and improved administration. Saleslogic e-Commerce: Cloudlogic SA’s Saleslogic is a consumer-facing e-commerce storefront designed to improve sales and drive productivity. Saleslogic lets you showcase and categorize products, implement custom designs and themes, and more. Semarchy SaaS – Solita Service powered by Azure: Solita Oy provides a hosted version of Semarchy xDM, enabling master data management powered by Microsoft Azure. This SaaS lets you implement value streams without concerns about infrastructure, security, or cloud operations. Senso for Teams Chat- Remote monitoring: Using an AI-based visual threat detection engine, Senso will monitor chat and inspect images in Microsoft Teams, alerting you when a violation is detected. Get Renato Software’s solution to make distance learning safer in your school, business, or district. SerendibAI: This AI-powered video analytics platform generates insights from available retail store video streams. The SerendibAI platform helps organizations make data-driven decisions to achieve business goals. SkyLIne ERP SaaS: SkyLIne by 2M Technologies is a complete ERP and management SaaS for manufacturers, distributors, and service companies. The Azure-based system provides visibility of sales, supply chains, finance, and more. Smart Connected Elevator: This solution from Larsen & Toubro Infotech uses IoT sensors to monitor elevator health, operations, and maintenance data, transforming field services. SmartWorkspaces for Windows Virtual Desktop: SmartWorkspaces simplifies Windows Virtual Desktop management and support by automating common manual processes. Streamline support, optimize user profile size, and perform remote assistance. Solteq Cloud POS: This cash register system can be deployed quickly to support omnichannel sales. Solteq Cloud POS supports traditional cash registers, mobile terminals, self-service kiosks, cash dispensers, and online stores. SOPHiA Platform: SOPHiA GENETICS’ platform unlocks health data through genomic and imaging data analysis for the management of cancer and rare diseases. SOPHiA allows clinical researchers to act with precision and confidence to improve health outcomes worldwide. SphereShield Ethical Wall for Skype for Business: AGAT Software Development LTD’s SphereShield Ethical Wall for Skype for Business lets you control who your Skype for Business users communicate with internally or externally. You can control or block specific options such as chat or file sharing. SphereShield MDM Integration for Skype for Business: The Mobile Device Management (MDM) Conditional Access Solution from AGAT Software Development LTD lets you ensure that users log in to Skype for Business only from devices managed by MDM. Survalyzer Survey Software: Survalyzer NG is a SaaS solution for creating, sending, analyzing, and visualizing surveys. Survalyzer includes unlimited respondents and works in more than 55 languages. Symphony for SAP: Basis Expert Consulting Services’ Symphony for SAP lets you orchestrate your SAP Basis and cloud operations on Microsoft Azure. You can automate actions across complex layers to increase efficiency and productivity. Syncier Cloud Cockpit: Cloud Cockpit from Syncier lets you centralize Kubernetes cluster management and automate compliance. Cloud Cockpit streamlines the management of multiple clusters and environments with a centralized platform, simplifying the deployment of applications via GitOps. Syncier GRNRY: Syncier’s GRNRY collects data in real time, optimizing the pipeline for explorative, analytical, and predictive use cases and event-driven integration scenarios for product development. Syncier Marketplace Public Area: Syncier offers a public marketplace for third-party APIs and software extensions relevant to the highly regulated insurance industry. Improve your offerings with digital services that integrate seamlessly into existing systems. SynCloud: SynCloud is a cloud economy cost management solution to optimize cloud spending across multiple cloud environments. SynCloud uses AI to overcome billing complexity. SysKit Point: SysKit Point is a cloud-based Microsoft 365 governance app made for administrators, IT professionals, and business managers to govern user access, security configuration, and content. The app lets you delegate governance and tasks, generate global security reports, and audit SharePoint. TradeEdge Market Connect: TradeEdge Market Connect is an automated two-way data exchange platform that enables the acquisition of sales, order, inventory, invoice, or similar information from channel partners, and it delivers the same information to manufacturers. Triofox Server: Triofox from Gladinet adds cloud enhancements to existing Windows file servers to address business concerns about security, privacy, compliance, and control. Keep data on-premises while making it available through an HTTPS-encrypted cloud layer. Ubuntu Server 18.04 LTS Minimal: This ready-to-launch image from Tidal Media includes a minimal version of Ubuntu Server 18.04 LTS. Ubuntu minimal images are optimized for automated use at scale, with a tiny package set and a reduced security vulnerability profile. Virtual Desktop Configurator for Media Production: Support Partners’ Creative Cloud VDI configurator lets you quickly customize and deploy a fully configured virtual environment in Microsoft Azure to allow you to manage Adobe Creative Cloud media production for your users. VisitorLAB: VisitorLab is a session-based web analytics service and tool that lets you see what your visitors see. Understand user behavior by tracking interactions, observing user movements, and detecting errors. Consulting services 1-Day App Innovation Assessment: Voice & KI: With a focus on product innovation, kuehlhaus will identify possibilities for voice interaction and AI using Microsoft Azure AI services. kuehlhaus will then deliver a summary of actions to pursue. This service is available only in German. 1-day assessment for existing solutions: In this assessment, kuehlhaus will identify optimization possibilities and provide a summary of recommendations and rearchitecting based on Microsoft Azure. This service is available only in German. Analytics Service – Build Data Platform: 2-Week Implementation: In this offer, B3IT Management will build and implement a cloud-based business intelligence solution using Microsoft Azure data services, then build an interactive Microsoft Power BI dashboard based on your data. Azure Backup Workshop – 1-day workshop: infoWAN’s workshop will introduce Microsoft Azure Backup, set up a backup job, create a backup report with Microsoft Power BI, and compose related email notifications. This service is available only in German. Azure Cloud Infrastructure Basics: 2-Day Workshop: Deep Network’s workshop will provide an overview of cloud computing and core Microsoft Azure services. Prerequisites include experience with files and directories, Git basics, basic coding skills, and an Azure subscription with a Contributor role. Azure Cloud Readiness: 4-Week Assessment: Looking to move to the cloud? Headspring’s assessment will determine your specific needs and devise a plan to equip you for success. Deliverables include a diagram of the proposed system architecture, a list of cloud migration options, and a migration cost analysis. Azure Cloud Strategy: 4-Week Assessment: In this assessment, Valence Group will look at your current challenges, processes, infrastructure, and goals, then provide a recommendation for Microsoft Azure services to fit your business strategy. Azure Cost Optimization: 10-Day Assessment: To optimize your Microsoft Azure spending, it’s essential to have a good understanding of available options and how they interact. In this engagement, CloudIBN will assess your infrastructure needs to improve your total cost of ownership. Azure DevOps Starter: 3-Day Workshop: NEOSYS will conduct this technical workshop via Microsoft Teams. The first day will involve Agile planning and configuring CI/CD pipelines. On the second day, NEOSYS will create an Azure Resource Manager template. The third day will focus on creating dashboards. Azure DevSecOps Service: 6-week assessment: Leaven, part of Computer Concepts Limited, will conduct a security review of your applications, Microsoft Azure subscriptions, and Azure DevOps to help you understand the security profile of your tenancy and apply fixes to vulnerabilities. Azure Infra. with Terraform: 1-Day Workshop: This workshop from Deep Network will teach you how to use HashiCorp Terraform to deploy Microsoft Azure infrastructure and infrastructure-as-code practices. Terraform is an open-source tool for provisioning and managing cloud infrastructure. Azure Serverless App Development: 2-Day Workshop: Over two days, Deep Network will show you the foundational concepts of serverless computing and how to use Microsoft Azure Functions. Workshop participants will be able to implement a complete message-processing system. Azure Synapse Analytics 2-Week Implementation: Microsoft Azure Synapse helps organizations build modern data architecture that enables efficient work with big data. This offer from CLOUD SERVICES will integrate data with data sources and deploy your migration data model to Azure Synapse. Azure- Windows Virtual Desktop (Native) 3-Day POC: In this proof of concept, SyCom will identify your business objectives and use Windows Virtual Desktop to deliver secure remote work capabilities. This offer includes consultation, implementation services, licensing, and monitoring. Azure- WVD (Citrix Cloud) 3-Day POC: In this proof of concept, SyCom will identify your business objectives and use Windows Virtual Desktop and Citrix Cloud to deliver secure remote work capabilities. This offer includes consultation, implementation services, licensing, and monitoring. AzWell – Azure Wellness check: 2-week assessment: This assessment from Eighty20 Solutions will focus on one key question: Are you using Microsoft Azure as well as you could be? Using various frameworks, Eighty20 Solutions will help your company optimize your Azure costs, security, and operations. Basic Cloud Cost Optimization: 1-Week Implementation: Rare Crew will assess your organization’s cloud infrastructure and identify potential cost leakages. Rare Crew will then implement real-time monitoring of Microsoft Azure resources and measure usage patterns to enable greater efficiency. CAF aligned Cloud Assessment & Migration: 10 Weeks: In this implementation service, Wipro’s Cloud Studio model, aligned with the Microsoft Cloud Adoption Framework for Azure, will help customers accelerate their cloud-led transformation. Customers will be able to plan, migrate, govern, and sustain varied workloads. Chat bots on Azure: 1-day workshop: Awara IT’s workshop will demonstrate how to use the Microsoft Bot Framework to create bots for business operations support. The workshop is intended for software developers with basic skills. Cloud Native WorkLab – 1-day workshop: In this workshop, prodot GmbH will create and document demand-driven solution architecture for your organization’s software. You’ll also receive a test deployment on your Microsoft Azure subscription. Cloud Service for Azure: Managed Infrastructure: With this managed service, Fujitsu Finland Oy will monitor your company’s Microsoft Azure infrastructure and workloads. You’ll benefit from quick adoption of new Azure features and consistent delivery across on-premises, hosted, cloud, and edge infrastructure. Cloud Service for Azure: Platform Support (Finland): Fujitsu Finland Oy’s platform support services for Microsoft Azure include extensive automation using Azure Lighthouse, straightforward Azure consumption-based pricing, and a clear line of support and escalation. Cloud Service for Azure: Platform Support (UK): Fujitsu Services United Kingdom’s platform support services for Microsoft Azure include extensive automation using Azure Lighthouse, straightforward Azure consumption-based pricing, and a clear line of support and escalation. Data Warehouse: 1-day assessment: Awara IT’s assessment, intended for banks and manufacturing companies, will enable clients to more efficiently use large amounts of data. Awara IT will evaluate the client’s environment to determine the best way to implement a data warehouse using Microsoft Azure services. Digital Strategy with Azure: 2-Day Workshop: Discover IT’s workshop will flesh out your company’s digital goals and compare them to your current digital landscape. Discover IT will consider any pain points and will propose solutions based on Microsoft Azure. Effective AKS and Logging: 3-Day Workshop: In this workshop, Deep Network will give an overview of managing logs inside an Azure Kubernetes Service cluster. Participants will then work with Elasticsearch and Kibana. There will also be time for some hands-on experience with Fluentd and Fluent Bit. GitHub Enterprise Adoption Jumpstart – 2-Week Implementation: Quisitive will help your organization adopt and implement GitHub Enterprise to manage and deliver custom solutions to Microsoft Azure. This will allow you to maximize developer velocity. IAM Assessment: 4-Week Assessment: This security and automation assessment from Innofactor, based on the Innofactor identity journey framework, will include an envisioning workshop, recommendations for Microsoft Azure Active Directory security and automation, and a roadmap for how to proceed. Innofactor Azure Starter pack – 5-week implementation: Over a period of five weeks, Innofactor will combine workshops and education with the delivery of a Microsoft Azure landing zone so your business can transition to a hybrid-cloud or cloud-only way of working. Microsoft SQL to Azure Migration: 3-Week Implementation: Migrate your Microsoft SQL database to Azure with the expertise of CloudIBN. CloudIBN will design a migration project plan, set up Azure infrastructure, oversee a pilot migration, and then conduct the live migration and handover. PUBSEC Cloud Adoption Framework: 2-Day Assessment: Sol-Tec’s assessment will benchmark your organization’s workloads against their current state and your aspirational priorities to identify any gaps in governance. Sol-Tec will help you understand your cloud adoption journey and will provide personalized recommendations to accelerate it. Sitecore on Azure PaaS – 1-Hour Assessment: Are you considering migrating your Sitecore solution to the cloud? Learn how Valtech and its global consulting and managed services for Sitecore PaaS on Microsoft Azure can deliver secure and flexible development, deployment, migration, and support. Syncier Managed Kubernetes: Unlock the full potential of your cloud infrastructure by automating container operations with Syncier’s multi-cloud Kubernetes-as-a-service solution. Syncier will adapt your Kubernetes environment to your IT infrastructure and offer personalized support. VDI Envisioning Workshop: 4-Hour Workshop: This envisioning workshop from Support Partners will discuss virtual desktop infrastructure and provide clients with a strategy to transition their creative workflows to Microsoft Azure. Veeam Cloud Connect Backup: 1-Day Implementation: PEAKUP will implement Veeam Cloud Connect, a secure and scalable cloud repository for Veeam Backup & Replication environments. Veeam Backup & Replication offers built-in cloud extensibility by using Microsoft Azure Blob storage accounts. Virtual Data Center: 2-Week Implementation: Innofactor Norway will implement a pre-configured Microsoft Azure landing zone of a virtual datacenter built with an infrastructure-as-code approach. The datacenter will be secure, governed by design, and implemented from Innofactor code libraries. Virtual Desktop: 2-Week Implementation: RAKE’s engineers will deliver a Windows Virtual Desktop environment to allow your employees to access everything they need to work from anywhere, anytime, and from any device. Windows Virtual Desktop: 2-Week Proof of Concept: Hexaware’s fully managed vWorkspace uses Windows Virtual Desktop to tackle remote work. This proof of concept allows users to test-drive Windows Virtual Desktop using their personal devices. Windows Virtual Desktop: 4-hour assessment: Awara IT aims to minimize IT infrastructure costs with this assessment, which will consider whether Windows Virtual Desktop is the right remote-work tool for your business. This offer is for companies employing at least 50 people. WorkZone: 5-Day Implementation: Neos IT will set up a secure and compliant virtual workplace for your company. This managed service includes virtual desktop infrastructure, licenses, subscriptions, Microsoft Azure, Microsoft 365, and Neos IT’s premium support.

Azure Redis Timeouts – Server Side Issues

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

There are many reasons that may cause timeouts on Redis client side, due to client, network or server side causes, and the error message also may differ based on Client library used.

Timeouts in Azure Cache for Redis occurs on client side when client application cannot receive the response from Redis server side timely.
Anytime, when client application doesn’t receive response before one of the timeout value expire, a timeout will occur and will be logged on client application logs as Redis timeout error message.

There are some causes on Azure Cache for Redis server or on the VM hosting it, that can also cause Redis timeouts on client side. If the cause of the delay is on Server side, the client application will not receive the response from Redis server timely and a timeout may happen.

Order by most common issues, below are the most common Server side causes: 

1- Server Update / Patch
2- Long Running Commands
3- Server High CPU Load

4- Server High Memory Usage

5- Redis Server Bandwidth Limit Exceeded

For Client or Network side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Client Side Issues
Azure Redis Timeouts – Network Issues

When any Redis Server update occurs, some Redis timeouts may happen on client side.
This is expected and by design as described below.

Standard/Premium Tiers of Azure Cache for Redis are made up of two Redis nodes. Each node running a single Redis server process and having a dedicated VM.

One instance is the “primary” node and other is the “replica” node. The primary node replicates data to the replica node more or less continuously.

If the primary node goes down unexpectedly, the replica will promote itself to primary, typically within 10-15 seconds. When the old primary node comes back up, it will become a replica and replicate data from the current primary node.

In the case of any update/patch, replica is proactively promoted to primary and the client should be able to immediately reconnect without any significant delay.

Basic tier cache has only one node, with no replica to take over during patching, so during update/patch the Basic tier Redis instances will go down and not be accessible during the patching process. 

This is one of the reasons why Basic caches are good for test/development environments but not recommended for production.

Redis updates:

On Azure Redis service, there are three types of updates:

• Redis updates/patches applied to the binaries used by Redis service;


• Host updates/patches applied to each Redis node VM.  The VM reboots after any Host update;


• Host OS updates/patches applied to operating system used by each Redis node VM. The VM reboots after any Host OS update.

If Azure Redis is in a Standard or Premium Tier, Redis Updates/patches, Host OS or Guest OS updates should not have any impact on the Redis availability.
 

Failover:

Every time a Redis Updates/patches occur an Redis failover is initiated.

When the failover occurs, the current connections on the primary node will be broken and will timeout on client application side – this is expected.

The number of errors seen by the client application depends on how many operations were pending on that connection at the time of the failover.

Any connection that’s routed through the node that closed its connections will see errors.

Many client libraries can throw different types of errors when connections break, including time-out exceptions, connection exceptions, or socket exceptions.

All the process is described in this documentation: How does a failover affect my client application?

How to verify Server Updates or Patch, from client side:

Currently there are no way to verify Redis, Host or Host OS updates/patches on client side or on Azure Portal. This is something currently in roadmap.

How to mitigate Server Updates or Patch:

As per Microsoft best practices, the application should retry to reconnect and when the retry happen, will reconnect to the new primary node without any impact on the service or to the end users.

Stackexchange.Redis:

Stackexchange.Redis have retry policy configured by default. With that some timeouts may be seen on client side, but with client retry policy, Redis service should be always available.

StackExchange.Redis client library has a setting named AbortConnect that controls how it handles connectivity errors like this. The default value for this setting is “True”, meaning that it will not reconnect automatically in some cases. Microsoft recommendation is to set AbortConnect to false, to let the ConnectionMultiplexer reconnect automatically.

Scheduling Redis updates:

Currently there’s feature available to allows schedule Redis updates. However, it does not cover platform updates (host or host OS updates), which happen much more often.

So at the moment, it’s not possible to specify the update time for all types of updates.
Please see this documentation showing how to Schedule updates

Some commands are more expensive than others to execute, depending on their complexity.

Because Redis is a single-threaded server side system, the time needed to run some more time expensive commands may cause some latency or timeouts on client side, as server can be busy dealing with these expensive commands.

Based on Redis.io documentation there are some well-known commands with higher time complexity, as described on top of each command description, on Redis.io page.

Time complexities:

O(1) – the command will be applied to only one specific key.

O(N) – with N being the number of keys in the database – this means the command will be applied to (or will read) each one of keys in the database – these commands should be avoided to use. When that is not possible, these type of commands should be used carefully.

These types of complex commands should be avoided, as described here:

https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-troubleshoot-server#long-running-commands

Also, Eval commands used to run LUA scripts on Redis side may need some time to finish.

Because Redis command processing is single-threaded, a command that takes time to run will block all others that come after.

How to verify Long Running Commands, from client side:

Redis-cli command line console can be used to run the SlowLog command, to measure expensive commands being executed against the server. This command is used to read or reset the Redis slow queries log, and the output is shown in microseconds. More information about SlowLog command: https://redis.io/commands/slowlog

Also, “Max CPU” and “Server Load” metrics on Azure Portal can be used to see how much server-side bandwidth is being used, usually corelated to long-running commands:

For that, Max aggregation should be used on the chart.
“Max CPU” and “Server Load” metrics are available to the Redis instance or per shard, in case of Redis clustered instances. On that case you may need to investigate each shard  “Max CPU” and “Server Load” usage to be able to identify any overload specifically in some shard.

How to mitigate Long Running Commands running:

To avoid having long running commands, time complex commands should be avoided and replaced by O(1) time complexity commands.

For instance, the KEYS command is often used without knowing that it’s an O(N) operation. Avoid KEYS can be done doing some more requests using SCAN O(1) to reduce CPU spikes.

Also, complex scripts should be reviewed to understand their performance impacts, and replaced by simple scripts. Having many simple scripts are preferable as having only few complex scripts.

High server load or CPU usage means the server can’t process requests in time. The server may be slow to respond and unable to keep up with request rates.

When server takes more time to respond as expected, this not means forcibly an issue on Server performance. Specially on Standard and Premium Redis tiers, Redis service have a dedicated VM with full resources available to Redis service, on each Redis node: primary and replica. 
If something wrong is detected on primary node, failover happen and the replica node can be promoted to new primary and take care of all work requested by client application.

Usually, server taking more time to respond than the expected are caused by client application requesting long-running commands or scripts, using large key values or using a cache size tier lower than the needed to deal with all the work requested by client application.
Also, high memory pressure can cause high number of page faults, and managing page faults can cause high CPU usage on Server side. Please see below on topic 4 how to investigate high memory pressure on Server side.

How to verify High Server CPU usage, from client side:

The “CPU” or “Server Load” metrics on Azure Portal can be used to find CPU usage and Server load.

For that, Max aggregation should be used on the chart.
“CPU” or “Server Load” metrics are available to the Redis instance or per shard, in case of Redis clustered instances. On that case you may need to investigate each shard individually to identify high “CPU” or “Server Load” in some specific shard.

How to mitigate High Server CPU usage:

There are several changes you may make to mitigate high server load:

• Investigate long running commands used as described above on topic 2.


• Scale to a larger cache size with more CPU capacity.


• Create alerts on metrics like CPU or server load to be notified early about potential impacts.

Memory pressure on the server side may cause some kinds of performance issues that can delay processing of requests, and when memory pressure occurs, the system may page data to disk.

This page faulting causes the system to slow down significantly.

There are some possible causes of this memory pressure:

• The cache is filled with data near its maximum capacity.


• Redis is seeing high memory fragmentation. This fragmentation is most often caused by storing large objects, since Redis is optimized for small objects.

How to verify High Server Memory usage, from client side:

• The “used_memory” and “used_memory_rss” metrics on Azure Portal can be used to see how much memory is being used:

For that, Max aggregation should be used on the chart.

• Any Redis console can be used to run Info command. This command may be used to identify memory usage looking for “Used Memory” and “Used Memory RSS“ section in the command output. Please see more about using Info command.

How to mitigate High Server Memory usage:

There are several possible changes to help keep memory usage healthy:

• Configure a memory policy and set expiration time on keys – this policy may not be sufficient if having fragmentation. 


• The default eviction policy is volatile-lru – this evict keys by trying to remove the less recently used (LRU) keys first, but only the keys that have an expire set. so, using volatile-lru, set expiration time on keys is highly recommended.


• Configure a maxmemory-reserved and maxfragmentationmemory-reserved value this should be large enough to compensate for memory fragmentation. Microsoft recommendation is to use 5% of the size of the cache (or higher) to each one of these reserved values, when using large key sizes.
  For the points above, please see how to Configure Memory Policies.


• Review client application logic and split large cached objects into smaller objects.


• Scale to a larger cache size with more memory capacity.


• Create alerts on metrics like used_memory and/or used_memory_rss to be notified early about potential impacts.

Redis Server Network Bandwidth exceeded and some other network causes for Redis timeouts are explained on this Tech Community article: Azure Redis Timeouts – Network Issues

Conclusion:

Despite having different reasons to have Redis timeouts on connections to Azure Cache for Redis, some different causes on Client, Server or Network side can cause Redis timeouts.

For Client or Network side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Client Side Issues
Azure Redis Timeouts – Network Issues

Related documentation:

Timeout issues 

Redis commands

How to Scale Azure Cache for Redis 
Creating Alerts on Azure

1- Server Update / Patch

How does a failover affect my client application?

Schedule updates

2- Long Running Commands

Slowlog

Long-running Commands

3- Server High CPU Load

High CPU or Server Load

4- Server High Memory Usage

Eviction policies
Configure Memory Policies

Memory pressure Server-side

Info command

I hope this can be useful !!!

Azure Redis Timeouts – Client Side Issues

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

There are many reasons that may cause timeouts on Redis client side, due to client, network or server side causes, and the error message also may differ based on Client library used. 

Timeouts in Azure Cache for Redis occurs on client side when client application cannot receive the response from Redis server side timely.
Anytime, when client application doesn’t receive response before one of the timeout values expire, a timeout will occur and will be logged on client application logs as Redis timeout error message.

As Redis service acts as a Memory cache for client applications, the performance on Redis calls should be high, with low response times. For that reason, timeout values used in the client application are usually low.
This means any stress or overload on client side will affect firstly Redis service, other than other applications, causing higher latency on Redis requests, and for that reason some Redis timeouts may occur. 

Order by most common issues, below are the most common Client side causes: 

1- Traffic Burst

2- Large Key Value Sizes

3- High Client CPU Load

4- Client Network Bandwidth Exceeded

5- Client Timeout Configurations

6- Idle Connections

7- High Client Memory Usage

For Network or Server side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Server Side Issues
Azure Redis Timeouts – Network Issues

Bursts of traffic combined with poor thread settings at client side can result in delays in processing data already sent by Redis Server but not yet consumed on the client side – this can cause Redis timeouts on client side.
Different client libraries can deal with this in different ways.

Traffic burst and ThreadPool settings on Stackexchange.Redis:

To deal with traffic burst and manage threadpool, Stackexchange.Redis client library have two types of threads:

• Worker threads: used for processing commands on Task.Run(…) or ThreadPool.QueueUserWorkItem(…) methods.
• IOCP (I/O Completion Port) threads: used when asynchronous IO happens (e.g. reading from the network).

Each type defines minimum number of threads and by default, the minimum is set to the number of cores in a client machine.
ThreadPool provides new threads on demand until it reaches a minimum.
When the number of IOCP or Worker used threads (Busy) reach the minimum (Min), the new threads will be throttled.
When throttle occurs, the rate at which it injects new threads is one thread per 500 milliseconds.
When client application gets burst of traffic that needs more threads than the minimum, then some threads will be throttled, and some timeouts may occur on Redis client.

Stackexchange.Redis exceptions contain useful information about a number of minimum and busy threads, for example:

In the above example, the number of busy Worker threads is 40 and the minimum is set to 2.
So, 38 threads * 500 ms = 19s; the 40th thread waited at least 19s to be created.
This exception has been thrown because Redis operation waited too long for a free thread.

Each application may need to adjust the “Min” value based on the usage and on the load.

How to monitor Traffic burst, from Client side:

– At network level, traffic bursts may be monitored by any network traffic analyzer to identify unexpected spikes on network usage;
– Stackexchange.Redis client library logs may be used to investigate higher “Busy” values than “Min”, on IOCP or Worker thread pools, as described above;
– ThreadPoolLogger application may be used to monitor and adjust “Min” values based on the “Busy” on bursts on traffic. 

https://github.com/JonCole/SampleCode/blob/master/ThreadPoolMonitor/ThreadPoolLogger.cs

How to adjust “Min” ThreadPool settings on Stackexchange.Redis:

There are two different ways to configure thread pool settings on Stackexchange.Redis:

1. Using SetMinThreads() on Redis StackExchange (C#) configuration (client side):

Thread pool settings on Stackexchange.Redis can be configured using SetMinThreads() on Redis StackExchange (C#) configuration, on client side application:

private readonly int minWorkerThreads = 200;
private readonly int minIOThreads = 200;
void Application_Start(object sender, EventArgs e)
{
    // Code that runs on application startup
   (…)
   ThreadPool.SetMinThreads(minWorkerThreads, minIOThreads);
}

Note: The value specified in when calling SetMinThreads IS NOT a per-core setting. For example, if you have a 4-core machine and want to set minWorkerThreads and minIoThreads to 200 per CPU during run-time, you should use ThreadPool.SetMinThreads(200, 200).

2. Using <processModel> on Microsoft Internet Information Services (IIS) web server:

It is also possible to specify the minimum threads setting by using the minIoThreads or minWorkerThreads configuration setting under the <processModel> configuration element in Machine.config, usually located at %SystemRoot%Microsoft.NETFramework[versionNumber]CONFIG.

<processModel>  on Machine.config

Setting the number of minimum threads in this way is generally not recommended, because it is a System-wide setting.

Note: The value specified in this configuration element is a per-core setting. For example, if you have a 4-core machine and want your minIoThreads setting to be 200 at runtime, you would use <processModel minIoThreads=”50″/>.

Redis cache is optimized to short key sizes (typical 1KB) and using larger key value sizes may cause Redis timeouts.

A large value size may be ok if making only few request at same time, but high number of requests with large key value sizes in a short time period can cause timeouts on Redis.

Because Redis is single-threaded system by design, processing any large key value will block all other requests that come after; if the first large key value takes more time than the expected, next requests will timeout.

In the following example, request ‘A’ and ‘B’ are sent quickly to the server. The server starts sending responses ‘A’ and ‘B’ quickly. Because of data transfer times, response ‘B’ must wait behind response ‘A’ times out even though the server responded quickly.

This is described in this documentation: Large request or response Size

Microsoft recommendation is to Optimize client application to use a large number of small values, rather than a few requests with large values.

In summary, large value sizes should be avoid and should be split in short key value sizes, making more requests.

How to verify Large Key Value Sizes used, from client side:

redis-cli.exe is a popular command-line tool for interacting with an Azure Cache for Redis as a client, and can be used to investigate large key value sizes used.

This tool is part of Redis-io distribution and also available for use with Azure Cache for Redis.

The only caveat of redis-cli.exe command-line tool is doesn’t support TLS and Azure Cache for Redis only enable the TLS port (6380) by default. 
To use this tool, an SSL/TLS access can be used using SSL tunnel, or temporarily enabling the non-TLS port (6379) (this is not recommended for security reasons, as the access keys are sent via TCP in clear text).
To use an SSL tunnel, Stunnel tool can be used and this Tech Community documentation may help on that: Enable access for redis-cli.exe

Download and extract redis-cli.exe command-line tool from Windows zip file for the latest Redis.io distribution, available here Redis-x64-3.0.504.zip

Then you can run redis-cli with this syntax (-p 6379 for non-SSL connections, or -p 6380 for SSL connections):

redis-cli -h <RedisName>.redis.cache.windows.net -p 6379 -a <RedisAccessPassword> --bigkeys

In this special mode, redis-cli works as a key space analyzer. It scans the dataset for big keys, but also provides information about the data types that the data set consists of.

This mode is enabled with the –bigkeys option, and produces quite a verbose output:

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest string found so far 'key-419' with 3 bytes
[05.14%] Biggest list   found so far 'mylist' with 100004 items
[35.77%] Biggest string found so far 'counter:__rand_int__' with 6 bytes
[73.91%] Biggest hash   found so far 'myobject' with 3 fields

-------- summary -------

Sampled 506 keys in the keyspace!
Total key length in bytes is 3452 (avg len 6.82)

Biggest string found 'counter:__rand_int__' has 6 bytes
Biggest   list found 'mylist' has 100004 items
Biggest   hash found 'myobject' has 3 fields

504 strings with 1403 bytes (99.60% of keys, avg size 2.78)
1 lists with 100004 items (00.20% of keys, avg size 100004.00)
0 sets with 0 members (00.00% of keys, avg size 0.00)
1 hashs with 3 fields (00.20% of keys, avg size 3.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)

More info about redic-cli and –bigkeys option can be consulted here: redis-cli, the Redis command line interface

How to avoid using Large Key Value Sizes:

To avoid using Large Key Value Sizes, Microsoft recommendation is to optimize client application for a large number of small values, rather than a few large values.

Large value sizes should be avoided and should be split in short key value sizes, making more requests.

High CPU usage on the client side can cause that the system cannot keep up with the work that it has been asked to perform. 

This could be a problem because if the CPU is busy, can’t keep up with the work the application is asking it to do.
The response from Redis can come very quickly, but because the CPU isn’t keeping up with the workload, the response sits in the client socket’s kernel buffer waiting to be processed.

Microsoft recommendation is not having Client CPU above 80%.

How to verify High Client CPU Load:

– Monitor the client’s system-wide CPU usage using metrics available in the Azure portal, in client environment blade (AppService, VM, AKS, etc);

– Monitor performance counters on client machine.

How to limit Client CPU usage:

– Investigate what is causing High CPU or CPU spikes on client side and modify the code logic.
Keep in mind that client CPU are usually not only used by Redis client application, but by other applications in the same AppService plan (in case of WebApps), or by other applications and operating system tasks, on VM’s, VMSS’s, AKS, etc.

– Upgrade your client to a larger VM size with more CPU capacity.

Client Network Bandwidth exceeded and some other network causes for Redis timeouts are explained on this Tech Community article: Azure Redis Timeouts – Network Issues

Each Redis client library has its own configuration options and the ability to configure different timeout values. Each option and the respective default value can be different based on the client library used.
As example, StackExchange.Redis uses the below related configuration options that can throws a timeout.

How to verify Redis Client Configurations:

To verify Redis Client Configurations you need to look for the connection string or the configuration object in the client application code, and check if the values used are the best for you case scenario and your workload.
Microsoft recommendation is to use only one connection and reuse connections.
Keep in mind that if not following Microsoft recommendations, client application may have different connection strings with different configuration values, in different parts of code. 

How to Configure Redis Client Timeout values:

StackExchange.Redis client library:
The connection to Azure Redis could be made through a connection string. Because there are lot of different ways to configure Redis, StackExchange.Redis offers a rich configuration model, which is invoked when calling Connect.

Specific to ConnectTimeout value on Stackexchange.Redis (the  amount of time used to the connection attempt), Microsoft recommends 15000ms, giving the system time to connect even under higher load conditions, but the default value is 5000ms. You may want to adjust this value.

The configuration can be either:

• a ConfigurationOptions instance:

string configString = "redis236.redis.cache.windows.net, abortConnect=false, ssl=true, password=XXX";
var options = ConfigurationOptions.Parse(configString);
options.AllowAdmin = true;
options.abortConnect=false;
options.connectTimeout=15000;

• a string representing the configuration:

string configString = "redis236.redis.cache.windows.net, abortConnect=false, ssl=true, password=XXX, connectTimeout=15000";

the switch between both configuration methods can be done trivially:

ConfigurationOptions options = ConfigurationOptions.Parse(configString);

or:

string configString = options.ToString();

Then use the configString or the ConfigurationOptions instance to create the connection with the new configuration values:

ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(configString);
ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(options);

Some other client library specific guidance:

• StackExchange.Redis (.NET)


• Java – Which client should I use?


• Lettuce (Java)


• Jedis (Java)


• Node.js


• PHP


• ASP.NET Session State Provider

Azure Cache for Redis service configures a 10 minute idle server timeout for connections, independently of the client library used. This is defined on Server side and cannot be changed. 
In that case, if the connection is idle for 10 minutes then it will reset the connection and clients may see socket reset errors, timeouts or some other connection error messages, depending of client libraries used.

Some clients like StackExchange.Redis will do regular PING calls to Redis to keep the connection alive (see keepAlive configuration above).
Some clients don’t do this, so application developers should implement this periodic ping call to avoid close idle connections.
This is discussed in Azure Redis Best Practices documentation. 

How to verify Idle Connections:

If you have idle connections without any Redis request during 10 minutes and you see socket reset error, timeout or some other connection error message, the most probably cause is not having any keep alive policy implemented.
You may also check for the specific client library used, if have any keepalive configuration option, as described for Stackexchange.Redis.

How to mitigate Idle Connections:

Adjust keepalive configuration option to 60 seconds (by default on Stackexchange.Redis), or the correspondent configuration value for different client libraries.
On some cases for some client libraries, you may need to implement your own keepalive process doing a periodic ping to Redis endpoint, to maintain the connation active.

Memory pressure on the client machine may be the cause of many kinds of performance problems that can delay processing of responses from the cache.
When this occur, the system may page data to disk, and page faulting causes the system to slow down significantly.

This may have higher impact on Redis calls, as is expected that Redis calls have high performance, with low response times. For that reason any memory pressure on client side will cause Redis timeouts prior than other services running on client environment.

How to verify High Client Memory Usage:

High Client Memory Usage can be verified at specific timestamp of Redis timeouts, monitoring memory usage on client machine to make sure that it doesn’t exceed available memory.

Could be useful also, Monitor the client’s Page Faults/Sec performance counter, as page faults can be caused by memory pressure.

How to limit Client Memory Usage:

Client Memory Usage can be reduced, reducing the number of running applications/processes at same time, or investigating the processes using higher amount of memory. If some process is using high amount of memory than the expected, this process/application should be troubleshooted to find the reason for that memory usage.

If all is expected, scaling the client environment to have more memory resources available should be implemented.

Conclusion:

Despite having different reasons to have Redis timeouts on connections to Azure Cache for Redis, some different causes on Client, Server or Network side can cause Redis timeouts, and some more details can be obtained in the documentation below.

For Network or Server side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Server Side Issues
Azure Redis Timeouts – Network Issues

Related documentation:

Timeout issues 

1- Traffic Burst

StackExchange.Redis timeout exceptions (GitHub article)  

ThreadPool Growth: Some Important Details (Jon Cole article)

Client-side issues        

BusyIO or BusyWorker threads in the timeout exception (GitHub discussion)    

ThreadPoolLogger code sample (from Jon Cole)

Traffic-burst

Recommendation ThreadPool settings

processModel Element

2- Large Key Value Sizes

redis-cli, the Redis command line interface

Enable access for redis-cli.exe

Redis-x64-3.0.504.zip

3- High Client CPU Load

High Client CPU usage

5- Client Timeout Configurations

Redis Client handling – TCP keepalive

Azure Redis Best Practices documentation 
StackExchange.Redis configurations

6- Idle Connections

Azure Redis Best Practices documentation

I hope this can be useful !!!

Azure Redis Timeouts – Network Issues

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

There are many reasons that may cause timeouts on Redis client side, due to client, network or server side causes, and the error message also may differ based on Client library used. 

Timeouts in Azure Cache for Redis occurs on client side when client application cannot receive the response from Redis server side timely.
Anytime, when client application doesn’t receive response before one of the timeout values expire, a timeout will occur and will be logged on client application logs as Redis timeout error message.

On Network, Redis timeouts may occur anytime due to connection failure, network blips, network failures, Client or Server Network bandwidth limit exceeded, or any other network issues may degrade Redis connectivity and may cause Redis timeouts on client side.

Order by most common issues, below are the most common Network causes: 

1- Network blips

2- Client Network bandwidth limit exceeded

3- Server Network bandwidth limit exceeded

For Client or Server side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Client Side Issues
Azure Redis Timeouts – Server Side Issues

Microsoft recommendation:

Microsoft recommendation is to use Azure Cache for Redis in the same Azure region as Redis client application.

The main reason for that is to take advantage of the large bandwidth used in the network backbone on each Azure region, with low latency, avoiding to use public networks with smaller bandwidth.

Azure Network backbone have many different ways to making it resilient to any failure.

In each Azure region, any eventual failure on Network backbone would affect all Azure services and all Customers, and not only the Azure Cache for Redis connections.

This is the main reason to avoid using Azure Cache for Redis service in a different Azure region than client application.

Network blips can be caused by Azure Load Balancer operations, failovers due to Redis or Host updates/patches, or some other reasons.

Because of the transient nature of the network blips, it’s not always possible to identify the cause of that network blips on client side. 

Network blips can be expected and Microsoft recommends to use some retry policy on client side application to deal with these transient network blips.
With that, network blips should be recovered by client application with a retry, avoiding any end user experience downtime.

StackExchange.Redis client library has a setting named AbortConnect that controls how it handles connectivity errors like this. The default value for this setting is “True”, meaning that it will not reconnect automatically in some cases. Microsoft recommendation is to set AbortConnect to false, to let the ConnectionMultiplexer reconnect automatically.

To identify if the reason for Redis timeouts are Client Network bandwidth limit exceeded, network usage on Client environment (AppService, VM, VMSS, AKS, etc) used by Redis Client application should be investigated, to see if is reaching any network interface limit or any network bandwidth limit (inbound or outbound).
Keep in mind that all applications in client side environment may have high network utilization and not only the Redis Client application; also, Redis timeouts can be caused not only by 100% network bandwidth utilization, but values reaching 100% can also cause higher network latency and Redis timeouts.
To troubleshoot this, some network tools on client side may needed, depending of client environment (AppService, VM, VMSS, AKS, etc).

To identify if the reason for Redis timeouts are caused by Server Network bandwidth limit exceeded, Azure portal have two important metrics on Redis service or Azure Monitor blades: Cache Read and Cache Write. These shows the amount of data read/written from/to the cache in Megabytes per second (MB/s), during any specified reporting interval: Available metrics and reporting intervals
Cache Read and Cache Write metrics are available to the Redis instance or per shard, in case of Redis clustered instances. On that case you may need to investigate each shard network usage to be able to identify any Network bandwidth limit exceeded specifically in some shard.
With that metrics, you can compare with the Azure Redis cache Network performance for your Redis tier: Azure Cache for Redis performance

Any other network causes for Redis network connectivity issues should be investigated on client side using some network trace to better understand where is the source of the problem.
A network packet trace from client side may help network team to identify the source of the problem.

Also, if needed specifically troubleshoot network connectivity, please see how to Troubleshooting Azure Redis Connectivity Issues (Tech Community article)

Conclusion:

Despite having different reasons to have Redis timeouts on connections to Azure Cache for Redis, some different causes on Client, Server side or Network can cause Redis timeouts.

For Client or Server side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Client Side Issues
Azure Redis Timeouts – Server Side Issues

Related documentation:

Timeout issues 
Redis Client handling – TCP keepalive

Best practices for Azure Cache for Redis (general and client library specific)

StackExchange.Redis best practices

Available metrics and reporting intervals

Azure Cache for Redis performance

Troubleshooting Azure Redis Connectivity Issues (Tech Community article)

I hope this can be useful !!!