Azure Redis Timeouts – Server Side Issues

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

There are many reasons that may cause timeouts on Redis client side, due to client, network or server side causes, and the error message also may differ based on Client library used.

Timeouts in Azure Cache for Redis occurs on client side when client application cannot receive the response from Redis server side timely.
Anytime, when client application doesn’t receive response before one of the timeout value expire, a timeout will occur and will be logged on client application logs as Redis timeout error message.

There are some causes on Azure Cache for Redis server or on the VM hosting it, that can also cause Redis timeouts on client side. If the cause of the delay is on Server side, the client application will not receive the response from Redis server timely and a timeout may happen.

Order by most common issues, below are the most common Server side causes: 

1- Server Update / Patch
2- Long Running Commands
3- Server High CPU Load

4- Server High Memory Usage

5- Redis Server Bandwidth Limit Exceeded

For Client or Network side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Client Side Issues
Azure Redis Timeouts – Network Issues

When any Redis Server update occurs, some Redis timeouts may happen on client side.
This is expected and by design as described below.

Standard/Premium Tiers of Azure Cache for Redis are made up of two Redis nodes. Each node running a single Redis server process and having a dedicated VM.

One instance is the “primary” node and other is the “replica” node. The primary node replicates data to the replica node more or less continuously.

If the primary node goes down unexpectedly, the replica will promote itself to primary, typically within 10-15 seconds. When the old primary node comes back up, it will become a replica and replicate data from the current primary node.

In the case of any update/patch, replica is proactively promoted to primary and the client should be able to immediately reconnect without any significant delay.

Basic tier cache has only one node, with no replica to take over during patching, so during update/patch the Basic tier Redis instances will go down and not be accessible during the patching process. 

This is one of the reasons why Basic caches are good for test/development environments but not recommended for production.

Redis updates:

On Azure Redis service, there are three types of updates:

• Redis updates/patches applied to the binaries used by Redis service;


• Host updates/patches applied to each Redis node VM.  The VM reboots after any Host update;


• Host OS updates/patches applied to operating system used by each Redis node VM. The VM reboots after any Host OS update.

If Azure Redis is in a Standard or Premium Tier, Redis Updates/patches, Host OS or Guest OS updates should not have any impact on the Redis availability.
 

Failover:

Every time a Redis Updates/patches occur an Redis failover is initiated.

When the failover occurs, the current connections on the primary node will be broken and will timeout on client application side – this is expected.

The number of errors seen by the client application depends on how many operations were pending on that connection at the time of the failover.

Any connection that’s routed through the node that closed its connections will see errors.

Many client libraries can throw different types of errors when connections break, including time-out exceptions, connection exceptions, or socket exceptions.

All the process is described in this documentation: How does a failover affect my client application?

How to verify Server Updates or Patch, from client side:

Currently there are no way to verify Redis, Host or Host OS updates/patches on client side or on Azure Portal. This is something currently in roadmap.

How to mitigate Server Updates or Patch:

As per Microsoft best practices, the application should retry to reconnect and when the retry happen, will reconnect to the new primary node without any impact on the service or to the end users.

Stackexchange.Redis:

Stackexchange.Redis have retry policy configured by default. With that some timeouts may be seen on client side, but with client retry policy, Redis service should be always available.

StackExchange.Redis client library has a setting named AbortConnect that controls how it handles connectivity errors like this. The default value for this setting is “True”, meaning that it will not reconnect automatically in some cases. Microsoft recommendation is to set AbortConnect to false, to let the ConnectionMultiplexer reconnect automatically.

Scheduling Redis updates:

Currently there’s feature available to allows schedule Redis updates. However, it does not cover platform updates (host or host OS updates), which happen much more often.

So at the moment, it’s not possible to specify the update time for all types of updates.
Please see this documentation showing how to Schedule updates

Some commands are more expensive than others to execute, depending on their complexity.

Because Redis is a single-threaded server side system, the time needed to run some more time expensive commands may cause some latency or timeouts on client side, as server can be busy dealing with these expensive commands.

Based on Redis.io documentation there are some well-known commands with higher time complexity, as described on top of each command description, on Redis.io page.

Time complexities:

O(1) – the command will be applied to only one specific key.

O(N) – with N being the number of keys in the database – this means the command will be applied to (or will read) each one of keys in the database – these commands should be avoided to use. When that is not possible, these type of commands should be used carefully.

These types of complex commands should be avoided, as described here:

https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-troubleshoot-server#long-running-commands

Also, Eval commands used to run LUA scripts on Redis side may need some time to finish.

Because Redis command processing is single-threaded, a command that takes time to run will block all others that come after.

How to verify Long Running Commands, from client side:

Redis-cli command line console can be used to run the SlowLog command, to measure expensive commands being executed against the server. This command is used to read or reset the Redis slow queries log, and the output is shown in microseconds. More information about SlowLog command: https://redis.io/commands/slowlog

Also, “Max CPU” and “Server Load” metrics on Azure Portal can be used to see how much server-side bandwidth is being used, usually corelated to long-running commands:

For that, Max aggregation should be used on the chart.
“Max CPU” and “Server Load” metrics are available to the Redis instance or per shard, in case of Redis clustered instances. On that case you may need to investigate each shard  “Max CPU” and “Server Load” usage to be able to identify any overload specifically in some shard.

How to mitigate Long Running Commands running:

To avoid having long running commands, time complex commands should be avoided and replaced by O(1) time complexity commands.

For instance, the KEYS command is often used without knowing that it’s an O(N) operation. Avoid KEYS can be done doing some more requests using SCAN O(1) to reduce CPU spikes.

Also, complex scripts should be reviewed to understand their performance impacts, and replaced by simple scripts. Having many simple scripts are preferable as having only few complex scripts.

High server load or CPU usage means the server can’t process requests in time. The server may be slow to respond and unable to keep up with request rates.

When server takes more time to respond as expected, this not means forcibly an issue on Server performance. Specially on Standard and Premium Redis tiers, Redis service have a dedicated VM with full resources available to Redis service, on each Redis node: primary and replica. 
If something wrong is detected on primary node, failover happen and the replica node can be promoted to new primary and take care of all work requested by client application.

Usually, server taking more time to respond than the expected are caused by client application requesting long-running commands or scripts, using large key values or using a cache size tier lower than the needed to deal with all the work requested by client application.
Also, high memory pressure can cause high number of page faults, and managing page faults can cause high CPU usage on Server side. Please see below on topic 4 how to investigate high memory pressure on Server side.

How to verify High Server CPU usage, from client side:

The “CPU” or “Server Load” metrics on Azure Portal can be used to find CPU usage and Server load.

For that, Max aggregation should be used on the chart.
“CPU” or “Server Load” metrics are available to the Redis instance or per shard, in case of Redis clustered instances. On that case you may need to investigate each shard individually to identify high “CPU” or “Server Load” in some specific shard.

How to mitigate High Server CPU usage:

There are several changes you may make to mitigate high server load:

• Investigate long running commands used as described above on topic 2.


• Scale to a larger cache size with more CPU capacity.


• Create alerts on metrics like CPU or server load to be notified early about potential impacts.

Memory pressure on the server side may cause some kinds of performance issues that can delay processing of requests, and when memory pressure occurs, the system may page data to disk.

This page faulting causes the system to slow down significantly.

There are some possible causes of this memory pressure:

• The cache is filled with data near its maximum capacity.


• Redis is seeing high memory fragmentation. This fragmentation is most often caused by storing large objects, since Redis is optimized for small objects.

How to verify High Server Memory usage, from client side:

• The “used_memory” and “used_memory_rss” metrics on Azure Portal can be used to see how much memory is being used:

For that, Max aggregation should be used on the chart.

• Any Redis console can be used to run Info command. This command may be used to identify memory usage looking for “Used Memory” and “Used Memory RSS“ section in the command output. Please see more about using Info command.

How to mitigate High Server Memory usage:

There are several possible changes to help keep memory usage healthy:

• Configure a memory policy and set expiration time on keys – this policy may not be sufficient if having fragmentation. 


• The default eviction policy is volatile-lru – this evict keys by trying to remove the less recently used (LRU) keys first, but only the keys that have an expire set. so, using volatile-lru, set expiration time on keys is highly recommended.


• Configure a maxmemory-reserved and maxfragmentationmemory-reserved value this should be large enough to compensate for memory fragmentation. Microsoft recommendation is to use 5% of the size of the cache (or higher) to each one of these reserved values, when using large key sizes.
  For the points above, please see how to Configure Memory Policies.


• Review client application logic and split large cached objects into smaller objects.


• Scale to a larger cache size with more memory capacity.


• Create alerts on metrics like used_memory and/or used_memory_rss to be notified early about potential impacts.

Redis Server Network Bandwidth exceeded and some other network causes for Redis timeouts are explained on this Tech Community article: Azure Redis Timeouts – Network Issues

Conclusion:

Despite having different reasons to have Redis timeouts on connections to Azure Cache for Redis, some different causes on Client, Server or Network side can cause Redis timeouts.

For Client or Network side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Client Side Issues
Azure Redis Timeouts – Network Issues

Related documentation:

Timeout issues 

Redis commands

How to Scale Azure Cache for Redis 
Creating Alerts on Azure

1- Server Update / Patch

How does a failover affect my client application?

Schedule updates

2- Long Running Commands

Slowlog

Long-running Commands

3- Server High CPU Load

High CPU or Server Load

4- Server High Memory Usage

Eviction policies
Configure Memory Policies

Memory pressure Server-side

Info command

I hope this can be useful !!!

Azure Redis Timeouts – Client Side Issues

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

There are many reasons that may cause timeouts on Redis client side, due to client, network or server side causes, and the error message also may differ based on Client library used. 

Timeouts in Azure Cache for Redis occurs on client side when client application cannot receive the response from Redis server side timely.
Anytime, when client application doesn’t receive response before one of the timeout values expire, a timeout will occur and will be logged on client application logs as Redis timeout error message.

As Redis service acts as a Memory cache for client applications, the performance on Redis calls should be high, with low response times. For that reason, timeout values used in the client application are usually low.
This means any stress or overload on client side will affect firstly Redis service, other than other applications, causing higher latency on Redis requests, and for that reason some Redis timeouts may occur. 

Order by most common issues, below are the most common Client side causes: 

1- Traffic Burst

2- Large Key Value Sizes

3- High Client CPU Load

4- Client Network Bandwidth Exceeded

5- Client Timeout Configurations

6- Idle Connections

7- High Client Memory Usage

For Network or Server side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Server Side Issues
Azure Redis Timeouts – Network Issues

Bursts of traffic combined with poor thread settings at client side can result in delays in processing data already sent by Redis Server but not yet consumed on the client side – this can cause Redis timeouts on client side.
Different client libraries can deal with this in different ways.

Traffic burst and ThreadPool settings on Stackexchange.Redis:

To deal with traffic burst and manage threadpool, Stackexchange.Redis client library have two types of threads:

• Worker threads: used for processing commands on Task.Run(…) or ThreadPool.QueueUserWorkItem(…) methods.
• IOCP (I/O Completion Port) threads: used when asynchronous IO happens (e.g. reading from the network).

Each type defines minimum number of threads and by default, the minimum is set to the number of cores in a client machine.
ThreadPool provides new threads on demand until it reaches a minimum.
When the number of IOCP or Worker used threads (Busy) reach the minimum (Min), the new threads will be throttled.
When throttle occurs, the rate at which it injects new threads is one thread per 500 milliseconds.
When client application gets burst of traffic that needs more threads than the minimum, then some threads will be throttled, and some timeouts may occur on Redis client.

Stackexchange.Redis exceptions contain useful information about a number of minimum and busy threads, for example:

In the above example, the number of busy Worker threads is 40 and the minimum is set to 2.
So, 38 threads * 500 ms = 19s; the 40th thread waited at least 19s to be created.
This exception has been thrown because Redis operation waited too long for a free thread.

Each application may need to adjust the “Min” value based on the usage and on the load.

How to monitor Traffic burst, from Client side:

– At network level, traffic bursts may be monitored by any network traffic analyzer to identify unexpected spikes on network usage;
– Stackexchange.Redis client library logs may be used to investigate higher “Busy” values than “Min”, on IOCP or Worker thread pools, as described above;
– ThreadPoolLogger application may be used to monitor and adjust “Min” values based on the “Busy” on bursts on traffic. 

https://github.com/JonCole/SampleCode/blob/master/ThreadPoolMonitor/ThreadPoolLogger.cs

How to adjust “Min” ThreadPool settings on Stackexchange.Redis:

There are two different ways to configure thread pool settings on Stackexchange.Redis:

1. Using SetMinThreads() on Redis StackExchange (C#) configuration (client side):

Thread pool settings on Stackexchange.Redis can be configured using SetMinThreads() on Redis StackExchange (C#) configuration, on client side application:

private readonly int minWorkerThreads = 200;
private readonly int minIOThreads = 200;
void Application_Start(object sender, EventArgs e)
{
    // Code that runs on application startup
   (…)
   ThreadPool.SetMinThreads(minWorkerThreads, minIOThreads);
}

Note: The value specified in when calling SetMinThreads IS NOT a per-core setting. For example, if you have a 4-core machine and want to set minWorkerThreads and minIoThreads to 200 per CPU during run-time, you should use ThreadPool.SetMinThreads(200, 200).

2. Using <processModel> on Microsoft Internet Information Services (IIS) web server:

It is also possible to specify the minimum threads setting by using the minIoThreads or minWorkerThreads configuration setting under the <processModel> configuration element in Machine.config, usually located at %SystemRoot%Microsoft.NETFramework[versionNumber]CONFIG.

<processModel>  on Machine.config

Setting the number of minimum threads in this way is generally not recommended, because it is a System-wide setting.

Note: The value specified in this configuration element is a per-core setting. For example, if you have a 4-core machine and want your minIoThreads setting to be 200 at runtime, you would use <processModel minIoThreads=”50″/>.

Redis cache is optimized to short key sizes (typical 1KB) and using larger key value sizes may cause Redis timeouts.

A large value size may be ok if making only few request at same time, but high number of requests with large key value sizes in a short time period can cause timeouts on Redis.

Because Redis is single-threaded system by design, processing any large key value will block all other requests that come after; if the first large key value takes more time than the expected, next requests will timeout.

In the following example, request ‘A’ and ‘B’ are sent quickly to the server. The server starts sending responses ‘A’ and ‘B’ quickly. Because of data transfer times, response ‘B’ must wait behind response ‘A’ times out even though the server responded quickly.

This is described in this documentation: Large request or response Size

Microsoft recommendation is to Optimize client application to use a large number of small values, rather than a few requests with large values.

In summary, large value sizes should be avoid and should be split in short key value sizes, making more requests.

How to verify Large Key Value Sizes used, from client side:

redis-cli.exe is a popular command-line tool for interacting with an Azure Cache for Redis as a client, and can be used to investigate large key value sizes used.

This tool is part of Redis-io distribution and also available for use with Azure Cache for Redis.

The only caveat of redis-cli.exe command-line tool is doesn’t support TLS and Azure Cache for Redis only enable the TLS port (6380) by default. 
To use this tool, an SSL/TLS access can be used using SSL tunnel, or temporarily enabling the non-TLS port (6379) (this is not recommended for security reasons, as the access keys are sent via TCP in clear text).
To use an SSL tunnel, Stunnel tool can be used and this Tech Community documentation may help on that: Enable access for redis-cli.exe

Download and extract redis-cli.exe command-line tool from Windows zip file for the latest Redis.io distribution, available here Redis-x64-3.0.504.zip

Then you can run redis-cli with this syntax (-p 6379 for non-SSL connections, or -p 6380 for SSL connections):

redis-cli -h <RedisName>.redis.cache.windows.net -p 6379 -a <RedisAccessPassword> --bigkeys

In this special mode, redis-cli works as a key space analyzer. It scans the dataset for big keys, but also provides information about the data types that the data set consists of.

This mode is enabled with the –bigkeys option, and produces quite a verbose output:

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest string found so far 'key-419' with 3 bytes
[05.14%] Biggest list   found so far 'mylist' with 100004 items
[35.77%] Biggest string found so far 'counter:__rand_int__' with 6 bytes
[73.91%] Biggest hash   found so far 'myobject' with 3 fields

-------- summary -------

Sampled 506 keys in the keyspace!
Total key length in bytes is 3452 (avg len 6.82)

Biggest string found 'counter:__rand_int__' has 6 bytes
Biggest   list found 'mylist' has 100004 items
Biggest   hash found 'myobject' has 3 fields

504 strings with 1403 bytes (99.60% of keys, avg size 2.78)
1 lists with 100004 items (00.20% of keys, avg size 100004.00)
0 sets with 0 members (00.00% of keys, avg size 0.00)
1 hashs with 3 fields (00.20% of keys, avg size 3.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)

More info about redic-cli and –bigkeys option can be consulted here: redis-cli, the Redis command line interface

How to avoid using Large Key Value Sizes:

To avoid using Large Key Value Sizes, Microsoft recommendation is to optimize client application for a large number of small values, rather than a few large values.

Large value sizes should be avoided and should be split in short key value sizes, making more requests.

High CPU usage on the client side can cause that the system cannot keep up with the work that it has been asked to perform. 

This could be a problem because if the CPU is busy, can’t keep up with the work the application is asking it to do.
The response from Redis can come very quickly, but because the CPU isn’t keeping up with the workload, the response sits in the client socket’s kernel buffer waiting to be processed.

Microsoft recommendation is not having Client CPU above 80%.

How to verify High Client CPU Load:

– Monitor the client’s system-wide CPU usage using metrics available in the Azure portal, in client environment blade (AppService, VM, AKS, etc);

– Monitor performance counters on client machine.

How to limit Client CPU usage:

– Investigate what is causing High CPU or CPU spikes on client side and modify the code logic.
Keep in mind that client CPU are usually not only used by Redis client application, but by other applications in the same AppService plan (in case of WebApps), or by other applications and operating system tasks, on VM’s, VMSS’s, AKS, etc.

– Upgrade your client to a larger VM size with more CPU capacity.

Client Network Bandwidth exceeded and some other network causes for Redis timeouts are explained on this Tech Community article: Azure Redis Timeouts – Network Issues

Each Redis client library has its own configuration options and the ability to configure different timeout values. Each option and the respective default value can be different based on the client library used.
As example, StackExchange.Redis uses the below related configuration options that can throws a timeout.

How to verify Redis Client Configurations:

To verify Redis Client Configurations you need to look for the connection string or the configuration object in the client application code, and check if the values used are the best for you case scenario and your workload.
Microsoft recommendation is to use only one connection and reuse connections.
Keep in mind that if not following Microsoft recommendations, client application may have different connection strings with different configuration values, in different parts of code. 

How to Configure Redis Client Timeout values:

StackExchange.Redis client library:
The connection to Azure Redis could be made through a connection string. Because there are lot of different ways to configure Redis, StackExchange.Redis offers a rich configuration model, which is invoked when calling Connect.

Specific to ConnectTimeout value on Stackexchange.Redis (the  amount of time used to the connection attempt), Microsoft recommends 15000ms, giving the system time to connect even under higher load conditions, but the default value is 5000ms. You may want to adjust this value.

The configuration can be either:

• a ConfigurationOptions instance:

string configString = "redis236.redis.cache.windows.net, abortConnect=false, ssl=true, password=XXX";
var options = ConfigurationOptions.Parse(configString);
options.AllowAdmin = true;
options.abortConnect=false;
options.connectTimeout=15000;

• a string representing the configuration:

string configString = "redis236.redis.cache.windows.net, abortConnect=false, ssl=true, password=XXX, connectTimeout=15000";

the switch between both configuration methods can be done trivially:

ConfigurationOptions options = ConfigurationOptions.Parse(configString);

or:

string configString = options.ToString();

Then use the configString or the ConfigurationOptions instance to create the connection with the new configuration values:

ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(configString);
ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(options);

Some other client library specific guidance:

• StackExchange.Redis (.NET)


• Java – Which client should I use?


• Lettuce (Java)


• Jedis (Java)


• Node.js


• PHP


• ASP.NET Session State Provider

Azure Cache for Redis service configures a 10 minute idle server timeout for connections, independently of the client library used. This is defined on Server side and cannot be changed. 
In that case, if the connection is idle for 10 minutes then it will reset the connection and clients may see socket reset errors, timeouts or some other connection error messages, depending of client libraries used.

Some clients like StackExchange.Redis will do regular PING calls to Redis to keep the connection alive (see keepAlive configuration above).
Some clients don’t do this, so application developers should implement this periodic ping call to avoid close idle connections.
This is discussed in Azure Redis Best Practices documentation. 

How to verify Idle Connections:

If you have idle connections without any Redis request during 10 minutes and you see socket reset error, timeout or some other connection error message, the most probably cause is not having any keep alive policy implemented.
You may also check for the specific client library used, if have any keepalive configuration option, as described for Stackexchange.Redis.

How to mitigate Idle Connections:

Adjust keepalive configuration option to 60 seconds (by default on Stackexchange.Redis), or the correspondent configuration value for different client libraries.
On some cases for some client libraries, you may need to implement your own keepalive process doing a periodic ping to Redis endpoint, to maintain the connation active.

Memory pressure on the client machine may be the cause of many kinds of performance problems that can delay processing of responses from the cache.
When this occur, the system may page data to disk, and page faulting causes the system to slow down significantly.

This may have higher impact on Redis calls, as is expected that Redis calls have high performance, with low response times. For that reason any memory pressure on client side will cause Redis timeouts prior than other services running on client environment.

How to verify High Client Memory Usage:

High Client Memory Usage can be verified at specific timestamp of Redis timeouts, monitoring memory usage on client machine to make sure that it doesn’t exceed available memory.

Could be useful also, Monitor the client’s Page Faults/Sec performance counter, as page faults can be caused by memory pressure.

How to limit Client Memory Usage:

Client Memory Usage can be reduced, reducing the number of running applications/processes at same time, or investigating the processes using higher amount of memory. If some process is using high amount of memory than the expected, this process/application should be troubleshooted to find the reason for that memory usage.

If all is expected, scaling the client environment to have more memory resources available should be implemented.

Conclusion:

Despite having different reasons to have Redis timeouts on connections to Azure Cache for Redis, some different causes on Client, Server or Network side can cause Redis timeouts, and some more details can be obtained in the documentation below.

For Network or Server side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Server Side Issues
Azure Redis Timeouts – Network Issues

Related documentation:

Timeout issues 

1- Traffic Burst

StackExchange.Redis timeout exceptions (GitHub article)  

ThreadPool Growth: Some Important Details (Jon Cole article)

Client-side issues        

BusyIO or BusyWorker threads in the timeout exception (GitHub discussion)    

ThreadPoolLogger code sample (from Jon Cole)

Traffic-burst

Recommendation ThreadPool settings

processModel Element

2- Large Key Value Sizes

redis-cli, the Redis command line interface

Enable access for redis-cli.exe

Redis-x64-3.0.504.zip

3- High Client CPU Load

High Client CPU usage

5- Client Timeout Configurations

Redis Client handling – TCP keepalive

Azure Redis Best Practices documentation 
StackExchange.Redis configurations

6- Idle Connections

Azure Redis Best Practices documentation

I hope this can be useful !!!

Azure Redis Timeouts – Network Issues

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

There are many reasons that may cause timeouts on Redis client side, due to client, network or server side causes, and the error message also may differ based on Client library used. 

Timeouts in Azure Cache for Redis occurs on client side when client application cannot receive the response from Redis server side timely.
Anytime, when client application doesn’t receive response before one of the timeout values expire, a timeout will occur and will be logged on client application logs as Redis timeout error message.

On Network, Redis timeouts may occur anytime due to connection failure, network blips, network failures, Client or Server Network bandwidth limit exceeded, or any other network issues may degrade Redis connectivity and may cause Redis timeouts on client side.

Order by most common issues, below are the most common Network causes: 

1- Network blips

2- Client Network bandwidth limit exceeded

3- Server Network bandwidth limit exceeded

For Client or Server side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Client Side Issues
Azure Redis Timeouts – Server Side Issues

Microsoft recommendation:

Microsoft recommendation is to use Azure Cache for Redis in the same Azure region as Redis client application.

The main reason for that is to take advantage of the large bandwidth used in the network backbone on each Azure region, with low latency, avoiding to use public networks with smaller bandwidth.

Azure Network backbone have many different ways to making it resilient to any failure.

In each Azure region, any eventual failure on Network backbone would affect all Azure services and all Customers, and not only the Azure Cache for Redis connections.

This is the main reason to avoid using Azure Cache for Redis service in a different Azure region than client application.

Network blips can be caused by Azure Load Balancer operations, failovers due to Redis or Host updates/patches, or some other reasons.

Because of the transient nature of the network blips, it’s not always possible to identify the cause of that network blips on client side. 

Network blips can be expected and Microsoft recommends to use some retry policy on client side application to deal with these transient network blips.
With that, network blips should be recovered by client application with a retry, avoiding any end user experience downtime.

StackExchange.Redis client library has a setting named AbortConnect that controls how it handles connectivity errors like this. The default value for this setting is “True”, meaning that it will not reconnect automatically in some cases. Microsoft recommendation is to set AbortConnect to false, to let the ConnectionMultiplexer reconnect automatically.

To identify if the reason for Redis timeouts are Client Network bandwidth limit exceeded, network usage on Client environment (AppService, VM, VMSS, AKS, etc) used by Redis Client application should be investigated, to see if is reaching any network interface limit or any network bandwidth limit (inbound or outbound).
Keep in mind that all applications in client side environment may have high network utilization and not only the Redis Client application; also, Redis timeouts can be caused not only by 100% network bandwidth utilization, but values reaching 100% can also cause higher network latency and Redis timeouts.
To troubleshoot this, some network tools on client side may needed, depending of client environment (AppService, VM, VMSS, AKS, etc).

To identify if the reason for Redis timeouts are caused by Server Network bandwidth limit exceeded, Azure portal have two important metrics on Redis service or Azure Monitor blades: Cache Read and Cache Write. These shows the amount of data read/written from/to the cache in Megabytes per second (MB/s), during any specified reporting interval: Available metrics and reporting intervals
Cache Read and Cache Write metrics are available to the Redis instance or per shard, in case of Redis clustered instances. On that case you may need to investigate each shard network usage to be able to identify any Network bandwidth limit exceeded specifically in some shard.
With that metrics, you can compare with the Azure Redis cache Network performance for your Redis tier: Azure Cache for Redis performance

Any other network causes for Redis network connectivity issues should be investigated on client side using some network trace to better understand where is the source of the problem.
A network packet trace from client side may help network team to identify the source of the problem.

Also, if needed specifically troubleshoot network connectivity, please see how to Troubleshooting Azure Redis Connectivity Issues (Tech Community article)

Conclusion:

Despite having different reasons to have Redis timeouts on connections to Azure Cache for Redis, some different causes on Client, Server side or Network can cause Redis timeouts.

For Client or Server side Redis timeout causes, please see these Tech Community articles:

Azure Redis Timeouts – Client Side Issues
Azure Redis Timeouts – Server Side Issues

Related documentation:

Timeout issues 
Redis Client handling – TCP keepalive

Best practices for Azure Cache for Redis (general and client library specific)

StackExchange.Redis best practices

Available metrics and reporting intervals

Azure Cache for Redis performance

Troubleshooting Azure Redis Connectivity Issues (Tech Community article)

I hope this can be useful !!!

Deploy SQL Server on Azure Kubernetes Service cluster via Helm Charts – on a windows client machine!

by Contributed | Jan 4, 2021 | Technology

This article is contributed. See the original author and article here.

A brief snapshot of the conversation that my colleague and I had a couple of weeks ago:

Colleague:  “I am new to Kubernetes and Linux world and was hoping if I could deploy SQL Server using helm chart from my Helm on my windows client and a managed kubernetes cluster service like Azure kubernetes service cluster (AKS)?”

Me:  “Yes, you can!”

Colleague: “Great! Can you share articles or guidance to do that so I could follow those?”

Me: “hmm.. For Helm refer this.. for Kubernetes refer this.. For SQL Server deployment using helm on kubernetes refer this.. “

and that’s when I noticed there wasn’t a single stop end to end guidance. This is what prompted me to write this blog. And I hope this helps anyone who is new to Kubernetes/helm charts or even SQL Server and want to get started with the deployment of SQL Server on Kubernetes.

By the end of this blog, you should have the SQL Server deployed via the helm charts on the managed kubernetes service which in this case is Azure Kubernetes Service (AKS). For more information on Kubernetes, Helm chart please refer to articles mentioned below to get you started.

1. What is Kubernetes ?


2. What is helm chart ?


3. The sample helm chart that we will use for this SQL Server deployment is available here. Please go through the readme file, to understand the files and values that you can chance and customize for your SQL Server deployment.

With the pre-read done, now lets start with deployment.

The follow along plan-

Please deploy an AKS (Azure Kubernetes Service) cluster.  If you do not want to use AKS and want to create your custom kubernetes cluster you could do that as well.

On your windows client system, from where you will connect to the AKS cluster and deploy SQL Server using the helm chart below, the tools that need to be installed on the windows client are:

• 
  

  You first install “az”, you can follow the steps documented here. From the link, download the current version of “az cli” exe and install the exe. If the installation is successful, you can type the command az in command prompt and if it is installed you should see output similar to the image shown below:

  
  

• Once you have the Azure CLI installed the next step is to install chocolatey, which is required to install helm on the windows client. To install chocolatey, follow the chocolatey install documentation. Follow the “Install with cmd.exe” section for a installation via the cmd.exe. Once the installation of the chocolatey is complete to confirm, open cmd.exe and run the command : choco as shown below:

• With chocolatey installed, it is now time to install helm. To install helm, open cmd.exe and run the command: 

  Choco install kubernetes-helm​

         Post the successful installation, when you run helm command in the cmd.exe you should see the output as shown below

• Now it is time to install the “kubectl” tool to interact and work with the kubernetes cluster, to install kubectl in the command prompt run the command

  az aks install-cli

         Post the successful install of “kubectl” you should add the path to “kubectl” as one of the  environment variables. In my case the path to kubectl is at “C:Usersamitkh.azure-kubectl” so I add the path to the environment variable as shown below:

With all the required tools installed on the windows client, we need to merge the context of the AKS cluster with the kubectl, so when we run kubectl or helm commands the operation takes place on that specific AKS cluster. To merge, run the command as described in the connect to aks cluster which is:

az aks get-credentials --resource-group <resourcegroupname> --name <aks clustername>

Once this is done, when you run a kubectl command the execution happens in the context of the aks cluster that we merged our tools with in the above step. An example shown below:

You are now ready to deploy the SQL Server on AKS cluster via the helm chart. You can download the sample helm-chart from this github location. Please go through the readme file to ensure you understand the options that you need or can change as per your requirement and customization.

Once you have the helm chart and all its file download to your windows client, switch to the directory where you have downloaded and after you have done modification to the downloaded helm chart to ensure it is as per your requirement and customization, deploy SQL Server using the command as shown below, you can change the deployment name (“mssql-latest-deploy”) to anything that you’d like.

helm install mssql-latest-deploy . --set ACCEPT_EULA.value=Y --set MSSQL_PID.value=Developer

Here is a example for reference, I have the chart and its files downloaded to the mssql-latest directory and this is how it looks and then I run the helm install command to deploy SQL Server.

After a few minutes, you should see SQL Server deployed and ready to be used as shown below:

Connect to the SQL Server running on AKS:

1. Using any of the familiar SQL Server tools like the SSMS (SQL Server Management Studio) or SQLCMD or ADS (Azure Data Studio), etc. You can connect to the instance of the SQL Server using the External-IP address for the “mssql-latest-deploy” service. In this case it is 20.40.0.145.


2. The sa_password is the value you provide to the Values.sa_password in the values.yaml file in the helm chart.

Changing the tempdb path:

As you would have noticed in the helm chart, we provide a specific location for tempdb files to ensure that the tempdb files are stored in those specific location. Here are the steps you can follow:

1. To change the tempdb location to the specific path you can connect to the SQL Server instance and then run the below T-SQL queries:

-- Get the tempdb specific files
select * from sysaltfiles where dbid=2
--We want to move the tempdb files to this specific location : /var/opt/mssql/tempdb/, ---so here are the commands:
ALTER DATABASE tempdb 
MODIFY FILE (NAME = tempdev, FILENAME = '/var/opt/mssql/tempdb/tempdb.mdf');
GO
ALTER DATABASE tempdb 
MODIFY FILE (NAME = tempdev2, FILENAME = '/var/opt/mssql/tempdb/tempdb2.ndf');
GO
ALTER DATABASE tempdb 
MODIFY FILE (NAME = templog, FILENAME = '/var/opt/mssql/tempdb/templog.ldf');
GO
--check and confirm that the locations for tempdb files are updated
select * from sysaltfiles where dbid=2

     2. Now for the change to take in affect, you will have to restart the SQL Server container and you can do the same using the commands below:

kubectl scale deployment mssql-latest-deploy --replicas=0

         Once you run the above command, wait for the container to be deleted and once it is, run the below command for the new container to be deployed

kubectl scale deployment mssql-latest-deploy --replicas=1

         Here you can now connect to the pod and check the tempdb files are now located to the new location : /var/opt/mssql/tempdb/

Hope you enjoyed it !! Happy learning!!

Amit Khandelwal

Sr. Program Manager