Announcement- System Center Data Protection Manager 2019 UR2 is here!

by Scott Muniz | Aug 10, 2020 | Uncategorized

This article is contributed. See the original author and article here.

System Center Data Protection Manager offers enterprise grade backup and recovery of critical workloads. We are excited to announce the release of Update Rollup 2 for System Center 2019 Data Protection Manager. This update rollup includes new features, critical bugs fixes and removes deployment blockers. This blog will cover the new features that are part of this release.

Support for SQL Server Failover Cluster Instance (FCI) using Cluster Shared Volume (CSV) 

A lot of SQL server users are looking to deploy SQL Server on a Failover Cluster instead of using a traditional cluster storage. This is advantageous for a couple of reasons: 

•        Users don’t need to carve out separate LUNs for each SQL instance.  
•        CSV makes SQL nodes highly available, resilient and fault tolerant by automatically routing traffic in case of node failure.  

Why backup is needed for SQL Server on CSV?

SQL Server on CSVs provides users with fault tolerance, high availability and resiliency, but it doesn’t guarantee that the data is safe in event of accidental deletion of data, data corruption, or ransomware attacks. In these scenarios, users would want a solution that they can use to restore their “clean” data from a previous point in time. 

DPM 2019 UR2 version introduces the support for protecting and recovering SQL Server Failover Cluster Instances on Cluster Shared Volumes. If you were already using SQL Server on CSVs, you can go ahead and start protecting your SQL DBs with DPM 2019 UR2. Learn more.

Optimized Migration for Backed Up Workloads

DPM 2019 supports a key customer requirement–migration of backed up workload data. There are several reasons why backed up workload migration is needed: 

•        Backups take long time due to high fragmentation on current  backup volume
•        The current backup volume could have reached the limit of maximum allowed storage size
•        The underlying storage box can have hardware issues and need to be retired

The visualization below explains how DPM currently implements backed up workload migration:

The above example represents a file which has 6 blocks of data – B1 to B6. The incremental recovery points store the changed blocks and metadata about unchanged blocks.

With this implementation of backed up workload migration, all the backup data (the full backup copy along with recovery points) is copied from the old backup volume to the new backup volume.

What’s new with DPM 2019 UR2 optimized migration? 

DPM 2019 UR2 introduces the concept of optimized migration, which allows you to move protected workloads from an old volume to a new volume in a much faster way. The enhanced migration process migrates only active backup copy (Active Replica) to the new volume. All the new recovery points are created on the new volume while existing recovery points are maintained on the existing volume. This less data transfer compared to the full volume migration approach results in a faster data source migration.

The visualization below explains how DPM implements optimized migration:

With this implementation of optimized migration for backup workload data, only the full backup copy is copied from the old backup volume to the new backup volume. As the old recovery points are kept on the existing volume and only the full backup copy is migrated to the new volume, the migration process finishes fast. For more information on how to enable this feature in DPM 2019 UR2, please refer to the documentation here.

SQL Server 2019 support as DPM database 

SQL Server 2019 is the latest version of SQL Server. Since a lot of SQL users have migrated to SQL 2019, DPM has also introduced support for SQL 2019 as its database starting DPM 2019 UR2 release.  

 Now you can install SQL Server 2019 on a remote server, or on the DPM server. The database must be installed and running before you install DPM. Learn more.

Offline Backup using Azure Data Box  

With DPM 2019 UR1 we announced the preview of offline seeding using Azure Data Box integration and now with UR2, this feature is open for any DPM customer to try out.  

With this integration, DPM customers can overcome the challenge of moving terabytes of backup data from on-premises to Azure storage. This also reduces the time a customer takes to onboard to Azure backup for long term retention of on-premises workload backups. This also reduces the time a customer takes to onboard to Azure backup for long-term retention of on-premises workload backups.

The below graphic illustrates how the offline seeding works for DPM customers looking to move terabytes of backup data to Azure storage. Learn More.

In addition, here is a link to the list of issues that have been fixed as part of the UR2 release for DPM. 

We hope you are as excited about the release of Update Rollup 2 for DPM 2019 as we are. We will continue to work on more updates and new features, and we would love to hear your feedback in the comments section below!

Ingesting log files from AWS S3 using AWS Lambda

by Scott Muniz | Aug 7, 2020 | Uncategorized

This article is contributed. See the original author and article here.

Do you have data going into S3 Buckets as files that you want to ingest into Azure Sentinel? 

In this blog, I will show you how to create an AWS Lambda function running PowerShell to ingest the data into Azure Sentinel.

Setup

To deploy this, you will need a machine prepared with the following:

• PowerShell Core – I recommend PowerShell 7 found here
• .Net Core 3.1 SDK found here
• AWSLambdaPSCore module – You can install this either from the PowerShell Gallery, or you can install it by using the following PowerShell Core shell command:
  • Install-Module AWSLambdaPSCore -Scope CurrentUser

See the documentation here https://docs.aws.amazon.com/lambda/latest/dg/powershell-devenv.html

I recommend you review https://docs.aws.amazon.com/lambda/latest/dg/powershell-package.html to review the cmdlets that are part of AWSLambdaPSCore.

Review of the Code

Our script will be triggered when a log file is created in an S3 bucket. To create the base script, run the following PowerShell core command:

New-AWSPowerShellLambda -Template S3Event

The default template for and S3Event trigger is below:

# PowerShell script file to be executed as a AWS Lambda function.
#
# When executing in Lambda the following variables will be predefined.
# $LambdaInput - A PSObject that contains the Lambda function input data.
# $LambdaContext - An Amazon.Lambda.Core.ILambdaContext object that contains information about the currently running Lambda environment.
#
# The last item in the PowerShell pipeline will be returned as the result of the Lambda function.
#
# To include PowerShell modules with your Lambda function, like the AWS.Tools.S3 module, add a "#Requires" statement
# indicating the module and version. If using an AWS.Tools.* module the AWS.Tools.Common module is also required.
#
# The following link contains documentation describing the structure of the S3 event object.
# https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html

#Requires -Modules @{ModuleName='AWS.Tools.Common';ModuleVersion='4.0.6.0'}
#Requires -Modules @{ModuleName='AWS.Tools.S3';ModuleVersion='4.0.6.0'}

# Uncomment to send the input event to CloudWatch Logs
# Write-Host (ConvertTo-Json -InputObject $LambdaInput -Compress -Depth 5)

foreach ($record in $LambdaInput.Records) {
  $bucket = $record.s3.bucket.name
  $key = $record.s3.object.key

  Write-Host "Processing event for: bucket = $bucket, key = $key"
  
  # TODO: Add logic to handle S3 event record, for example
  $obj = Get-S3Object -Bucket $bucket -Key $key
  Write-Host "Object $key is $($obj.Size) bytes"
}

As you can see for each CreateObject event from S3 the default script pulls the bucket name ($bucket = $record.s3.bucket.name) and filename ($key = $record.s3.object.key) Then Write-Host will send an output to CloudWatch logs.  Lastly it gets the file size using the Get-S3object cmdlet.

I used this basic template as a starter template.  First, I added a function called Write-OMSLogFile which I got from Travis Roberts on GitHub here.  In short, the function takes a hash table with the log data and a few parameters and uploads it to the Log Analytics REST API.  The documentation on this API is here.  The function needs three parameters you will need to provide:

• Line 158 – $WorkspaceId – you will need to enter the workspace id
• Line 159 – $WorkspaceKey – you will need to enter the workspace key
• Line 160 – $CustomLogName – you will need to enter the Custom Log name you want to use for the data.

 After we have those setup, the first action we need to do is download the file from S3 to tmp storage.

Write-Host "Downloading $key to /tmp/$key"
Read-S3Object -BucketName $bucket -Key $key -File "/tmp/$key"
Write-Host "Downloaded $key to /tmp/$key"

Once we have the file, we need to process the file based on file type.  I have made some simple assumptions.

• If its .csv its CSV data which is easy to import to a hash table using Import-CSV.
• If its .json, its JSON data which is easy to import to hash table using Get-Content | ConvertFrom-JSON
• Lastly, if it’s a .log file the data contains CEF formatted data which takes some special processing.
  • I used some REGEX and splitting to break apart the CEF Message and then build a hash table of records.
• Or its not supported in this function yet.

#Determine if CSV or JSON or whatever
$FileName = "/tmp/$key"
if ($fileName -like "*.csv") {
  Write-Host "Handling CSV File"
  $data = import-csv $filename
}
elseif ($filename -like "*.json") {
  Write-Host "Handling JSON File"
  $Data = Get-Content $filename | ConvertFrom-Json
}
elseif ($filename -like "*.log") {
  Write-Host "Handling Log File"
  #Assuming CEF formatted logs
  $cefdata = Get-Content $filename
  $data = @()
  $cefmsg = @{}
  foreach ($line in $cefdata) {
    if ($line -like "*CEF:*") {
      #Write-Host "Handling CEF Data"
      $CEFtimegenerated = ($line -split '(?<time>(?:w+ +){2,3}(?:d+:){2}d+|d{4}-d{2}-d{2}Td{2}:d{2}:d{2}.[w-:+]{3,12})')[1]
      #$CEFHost = (($line -split '(?<time>(?:w+ +){2,3}(?:d+:){2}d+|d{4}-d{2}-d{2}Td{2}:d{2}:d{2}.[w-:+]{3,12})')[2] -split "CEF:")[0]
      #$CEFVersion = $line.Split("CEF: ").Split("|")[1]
      $CEFDeviceVendor = $line.split("|")[1]
      $CEFDeviceProduct = $line.split("|")[2]
      $CEFDeviceVersion = $line.split("|")[3]
      $CEFDeviceEventClassId = $line.split("|")[4]
      $CEFName = $line.split("|")[5]
      $CEFSeverity = $line.split("|")[6]
      $CEFExtension = $line.split("|")[7] -split '([^=s]+=(?:[]=|[^=])+)(?:s|$)'
      foreach ($extenstion in $CEFExtension) {
        if ($extenstion -like "*=*") { $cefmsg += @{$extenstion.Split("=")[0] = $extenstion.Split("=")[1] } }
      }
      $CEFmsg += @{TimeGenerated = $CEFtimegenerated }
      $CEFmsg += @{DeviceVendor = $CEFDeviceVendor }
      $CEFmsg += @{DeviceProduct = $CEFDeviceProduct }
      $CEFmsg += @{DeviceVersion = $CEFDeviceVersion }
      $CEFmsg += @{DeviceEventClassID = $CEFDeviceEventClassId }
      $CEFmsg += @{Activity = $CEFName }
      $CEFmsg += @{LogSeverity = $CEFSeverity }
      $data += $CEFmsg
      $cefmsg = @{}
    }
  }
  Write-Host "Finished Handling Log file"
}
else { Write-Host "$filename is not supported yet" }

Next we can upload the data to Log Analytics.  The API supports up to 30MB per upload.  Since FileSize is not equal to HashTable size which is normally less than JSON actual size, we need to convert the data to JSON and measure the actual size.  I created a loop to add each record to a temp object until that object is greater than 25MB then upload that chunk and start over. I used 25MB as a safety.  If its less than 25MB to start with then just upload the data as is.

#Test Size; Log A limit is 30MB
$tempdata = @()
$tempDataSize = 0
Write-Host "Checking if upload is over 25MB"
if ((($Data | Convertto-json -depth 20).Length) -gt 25MB) {
  Write-Host "Upload needs to be split"
  foreach ($record in $data) {
    $tempdata += $record
    $tempDataSize += ($record | ConvertTo-Json -depth 20).Length
    if ($tempDataSize -gt 25MB) {
       Write-OMSLogfile -dateTime (Get-Date) -type $CustomLogName -logdata $tempdata -CustomerID $workspaceId -SharedKey $workspaceKey
       Write-Host "Sending data = $TempDataSize"
       $tempdata = $null
       $tempdata = @()
       $tempDataSize = 0
    }
  }
  Write-Host "Sending left over data = $Tempdatasize"
  Write-OMSLogfile -dateTime (Get-Date) -type $CustomLogName -logdata $tempdata -CustomerID $workspaceId -SharedKey $workspaceKey
}
Else {
  #Send to Log A as is
  Write-Host "Upload does not need to be split, sending to Log A"
  Write-OMSLogfile -dateTime (Get-Date) -type $CustomLogName -logdata $Data -CustomerID $workspaceId -SharedKey $workspaceKey
}

Lastly we need to clean up the file we downloaded.

Remove-Item $FileName -Force

Deploy the code

The Lambda function will need an execution role defined that grants access to the S3 bucket and CloudWatch logs.  To create an execution role:

1. Open the roles page in the IAM console.
2. Choose Create role.
3. Create a role with the following properties.
   • Trusted entity – AWS Lambda.
   • Permissions – AWSLambdaExecute.
   • Role name – lambda-s3-role.

The AWSLambdaExecute policy has the permissions that the function needs to manage objects in Amazon S3 and write logs to CloudWatch Logs.  Copy the Amazon Resource Name (ARN) of the role created as you will need it for the next step.

To deploy the PowerShell script, you can create a Package (zip file) to upload to the AWS console or you can use the Publish-AWSPowerShell cmdlet.   We will use the cmdlet.

Publish-AWSPowerShellLambda -Name YourLambdaNameHere -ScriptPath <path>/S3Event.ps1 -Region <region> -IAMRoleArn <arn of role created earlier> -ProfileName <profile>

You might need –ProfileName if your configuration of .aws/credentials file doesn’t contain a default.  See this document for information on setting up your AWS credentials.

Here is the sample output from my running of the cmdlet.

…....
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/Microsoft.WSMan.Management/Microsoft.WSMan.Management.psd1 (deflated 42%)
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/CimCmdlets/CimCmdlets.psd1 (deflated 44%)
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Diagnostics/Microsoft.PowerShell.Diagnostics.psd1 (deflated 42%)
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Diagnostics/Event.format.ps1xml (deflated 58%)
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Diagnostics/Diagnostics.format.ps1xml (deflated 54%)
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Diagnostics/GetEvent.types.ps1xml (deflated 55%)
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Security/Microsoft.PowerShell.Security.psd1 (deflated 42%)
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/PSDiagnostics/PSDiagnostics.psm1 (deflated 59%)
... zipping:   adding: runtimes/win/lib/netcoreapp3.1/Modules/PSDiagnostics/PSDiagnostics.psd1 (deflated 42%)
... zipping:   adding: runtimes/unix/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Utility/Microsoft.PowerShell.Utility.psd1 (deflated 59%)
... zipping:   adding: runtimes/unix/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Management/Microsoft.PowerShell.Management.psd1 (deflated 60%)
... zipping:   adding: runtimes/unix/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Host/Microsoft.PowerShell.Host.psd1 (deflated 35%)
... zipping:   adding: runtimes/unix/lib/netcoreapp3.1/Modules/Microsoft.PowerShell.Security/Microsoft.PowerShell.Security.psd1 (deflated 39%)
Created publish archive (/private/var/folders/hv/786wbs4n13ldvz9765nx8nv40000gn/T/S3toSentinel/bin/Release/netcoreapp3.1/S3toSentinel.zip).
Creating new Lambda function S3toSentinel
New Lambda function created

Once created, login to the AWS console.   In Find services, search for Lambda.  Click on Lambda.

Click on the lambda function name you used with the cmdlet. Click Add Trigger

Select S3.  Select the bucket.  Acknowledge the warning at the bottom.  Click Add.

Your lambda function is ready to send data to Log Analytics.  

Test the Code

To test your function, upload a support file to the S3 bucket defined in the trigger.

To view the results of your test, go the Lambda function.  Click Monitoring tab.  Click view logs in CloudWatch.

In CloudWatch, you will see each log stream from the runs.  Select the latest.  Here you can see anything from the script from the Write-Host cmdlet.

Go to portal.azure.com and verify your data is in the custom log.  Run a query for your <CustomLogName>_CL.  If this is a new custom log it may take a few minutes for results to show up.  For more on queries see https://docs.microsoft.com/azure/azure-monitor/log-query/get-started-queries

NOTE: To process large files, you may need to increase the Lambda function timeout.  For more information on timeouts see https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html 

Next Steps:

• Create an analytic rule on your data – https://docs.microsoft.com/azure/sentinel/tutorial-detect-threats-custom
• Create a workbook on your data – https://docs.microsoft.com/en-us/azure/sentinel/tutorial-monitor-your-data

I hope this will help some folks solve a unique challenge!  You can find the full code and README here.

Thanks to @Chi_Nguyen, Joel Stidley, and @sarahyo  for the help on this.

Best,

Nicholas DiCola

Security Jedi

Event ID 7023: The Windows Process Activation Service terminated

by Scott Muniz | Aug 7, 2020 | Uncategorized

This article is contributed. See the original author and article here.

If your web applications stop responding requests, WAS is one of the services you should check first.

While troubleshooting an issue in a web server, I saw this error in the Event Viewer:

Event ID 7023: The Windows Process Activation Service service terminated with the following error: ​The system cannot find the file specified

Solution

Follow the steps below to fix this issue:

1. Check c:/windows/system32/inetsrv/ folder. applicationHost.config file might be missing or it might be empty. If it is, go to c:/inetpub/history/ folder and copy the applicationHost.config file from there to  inetsrv folder. Try to start Windows Process Activation Service again
2. Go to c:inetpubtemp folder. Check if there is an apppools folder. If it doesn’t exist, create this folder. Try starting WAS again
3. Open registry editor. Navigate to HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesWASParameters. Delete NanoSetup entry. Try again
4. Go inetsrv folder. Duplicate applicationHost.config file. Name the new file applicationHost.config.tmp. Try to start Windows Process Activation Service again

Web Deploy issue ERROR_CONNECTION_TERMINATED

by Scott Muniz | Aug 7, 2020 | Uncategorized

This article is contributed. See the original author and article here.

While publishing an application from Visual Studio to Azure App Service, you may come across this error message:

Web Deploy experienced a connection problem with the server and had to terminate the connection.

Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED

In the case I worked on, the project publishes successfully to Local IIS and Azure Functions but not to Azure App Service.

Solution

Follow the steps below to find a solution to this issue:

• Uninstall security software in the server and try again
• Make sure Management Services are installed (Add Role and Features > Web Server)
• Check if Web Management Services and Web Deploy Agent Service are running
• Make sure Fiddler or any other similar software is not running during the publish
• Check if TCP port 8172 are open

Timer_MinBytesPerSecond in HTTP.SYS logs

by Scott Muniz | Aug 7, 2020 | Uncategorized

This article is contributed. See the original author and article here.

In order to troubleshoot slowness in web, system administrators should analyze IIS logs, Failed Request Tracing logs, dump files, and HTTP.SYS logs to narrow down the issue.

In a slowness issue I worked on, these records in HTTPERR file took my attention:

2020-07-26 14:09:01 10.123.12.12 35316 10.12.123.123 80 - - - - - Timer_MinBytesPerSecond
2020-07-26 14:09:01 10.123.12.12 48012 10.12.123.123 80 - - - - - Timer_MinBytesPerSecond

Apparently, requests take more than 35 seconds to response due to a Timer_MinBytesPerSecond related issue.

What is MinBytesPerSecond?

It’s a parameter HTTP.SYS uses to determine what speed is allowed for the response to the client.

From the official document:

Specifies the minimum throughput rate, in bytes, that HTTP.sys enforces when it sends a response to the client. The minBytesPerSecond attribute prevents malicious or malfunctioning software clients from using resources by holding a connection open with minimal data. If the throughput rate is lower than the minBytesPerSecond setting, the connection is terminated.

The current implementation only terminates the connection after the time it would have taken to stream the entire response back to the connecting client at the minimum bandwidth transfer rate has elapsed. If the transfer rate goes below the value specified by minBytesPerSecond only for a small period of time, but the overall transfer rate is higher, the connection will not be terminated.

The default value is 240.

Solution

HTTP.SYS logs above shows that the root cause of the issue is the slow response throughput. The client is not receiving IIS response at a reasonable speed.

For Windows Server 2012 R2 (it was the server I worked on), IIS wants to send response at minimum 240 bytes per second. If the client cannot receive it at this minimum speed, IIS terminates the connection. This logic prevents the server against denial-of-service (DoS) attacks.

1. Find out which clients are causing this issue

• The client machines might be in a slow network. If the issue is caused by the same client(s), you may be able to improve their condition to solve the issue
• Find out client IP addresses in the HTTPERR file and investigate their network as well as their OS

• If the client IPs are legit (not malicious sources or possible attackers), set minBytesPerSecond to 0 
• If the issue doesn’t happen again, play around the value (Try 500). It’s not recommended to set it 0 because it may make your application vulnerable to DoS attacks
• Steps for changing minBytesPerSecond:

1. Go to IIS Manager
2. Click on the server name
3. Double click on Configuration Editor
4. Go to “system.applicationHost/webLimits”
5. Configure minBytesPerSecond
6. Reset IIS