by Contributed | Jan 16, 2021 | Technology
This article is contributed. See the original author and article here.
As Synapse engineer or Synapse Support Engineer you may need to start and test some Pools, and you want this to be the most cost efficient possible. Leaving some Synapse with a lot of DWU left turned on during the weekend because you forget to pause the DW after you shutdown your computers is not a good approach and we can quickly resolve this by using Powershell + Automation accounts.
Before we get into the procedure on the automation, just want to show you some details.
We currently have two flavors of Synapse
- Dedicated SQL pools (formerly SQL DW)
- Old mode where the SQL DW lives in an Azure SQL DB Server that can be shared with regular Azure SQL databases.
- Internally is a resource of type (Microsoft.SQL)
- Sample: /subscriptions/xxxxxx/resourceGroups/yyyyyyy/providers/Microsoft.Sql/servers/yyyyyyyy/databases/olddwpool
- Azure Synapse Analytics – Dedicated SQL pool
- SQL DW database inside a workspace
- Internally is a resource of type (Microsoft.Synapse)
- Sample: /subscriptions/xxxxxx/resourceGroups/yyyyyyy/providers/Microsoft.Synapse/workspaces/yyyyyyyy/sqlPools/dwpool
This is important to know because we are going to use 2 different scripts to pause the pool
- Suspend-AzSynapseSqlPool (Az.Synapse)
- Suspend-AzSqlDatabase (Az.SQL)
There are 2 versions of the script
- Powershell to run from your machine
- Using Azure Automation Account
1. Powershell to run from your machine
This script you can run from your machine and you can get last version of script at ScriptCollection/Synapse – Pause all DWs.ps1 at master · FonsecaSergio/ScriptCollection · GitHub
You are going to need the modules installed
- Az.Accounts
- Az.Sql
- Az.Synapse
For below script I’m considering that your user is same that you use to admin your Azure Subscription
Find a sample below
Context exists
Current credential is sefonsec@microsoft.com
Current subscription is SEFONSEC Microsoft Azure Internal Consumption
—————————————————————————————————
Get SQL / Synapse RESOURCES
—————————————————————————————————
—————————————————————————————————
Loop through all Synapse Workspaces
—————————————————————————————————
Checking Azure Synapse Workspace [xxxxxxxxxxx_synapse] for Synapse SQL Pools
-> Synapse SQL Pool [dwpool] found with status [Online]
-> Pausing Synapse SQL Pool [dwpool]
-> Synapse SQL Pool [dwpool] paused in 0 hours, 2 minutes and 32 seconds. Current status [Paused]
—————————————————————————————————
Loop through all SQL Servers (former SQLDW)
—————————————————————————————————
Checking SQL Server [xxxxxxxxxxx-eastus] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-eastus2] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-northeu] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-southcentralus] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-uksouth] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-ukwest] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx] in Resource Group [CSSAzureDB] for Synapse SQL Pools
-> Synapse SQL Pool [SQLDW] found with status [Paused]
Checking SQL Server [xxxxxxxxxxx-byok] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-demo] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx_synapse] in Resource Group [synapseworkspace-managedrg-5da694c3-ae72-4f25-9cc6-626adcf858e6] for Synapse SQL Pools
-> This DB is part of Synapse Workspace – Ignore here Should be done above using Az.Synapse Module
Checking SQL Server [xxxxxxxxxxx-westeu] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-westus] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
A sample with error
Checking Azure Synapse Workspace [xxxxxxxxxxx_synapse] for Synapse SQL Pools
Write-Error: -> Checking Synapse SQL Pool [dwpool] found with status [Resuming]
2. Using Azure Automation Account
Now we want this to be automated, like as a sample to shutdown every day at 11PM and send me alerts if error happens
You can find last version at: ScriptCollection/Synapse – Pause all DWs – Automation Acount.ps1 at master · FonsecaSergio/ScriptCollection · GitHub
1 – Lets first create the Automation Account
2 – Make sure to create a Run As Account
3 – By default it already got the contributor permission in the subscription level. You can change that if needed. Or add the any required permissions.
4 – You need to install the Az modules. Just go to modules and look for them in the gallery.
5 – Install first the Az.Account because it’s a prerequisite for others
Wait for it to complete
6 – Do same for 3 modules
- Az.Accounts
- Az.Sql
- Az.Synapse
7 – Now go to runbooks and create a new Powershell runbook
8 – Add code from ScriptCollection/Synapse – Pause all DWs – Automation Acount.ps1 at master · FonsecaSergio/ScriptCollection · GitHub
8.1 – Save and Publish. And click on Start to test it
9 – Check output tab
10 – You can now go to schedules and add a new schedule
11 – You may also want also to be alerted if schedule fail. Just to back to automation account and click on Alerts
12 – You need to add a condition when alert will fire and what actions will be taken. This action could be email to you or run some process or run another script
13 – First configure the condition. You are going to use the metric Total Jobs (Like total jobs failed)
14 – Define
– runbook name
– status (Add custom status “Failed“)
– Threshold > 0
– Run every hour
15 – Add now the action group. In this case, send email to me
You are now set. So your DW will be stopped at 11PM if some error happen you will be alerted like as a sample when a DW is in a state that could not be paused.
Another last tip is to use the Azure Budget control I got from my colleague @Gonçalo Ventura
Create a budget and alert
The budget puts a maximum limit on the cost of the subscription, in case some service is left running the subscription will automatically suspend when the budget is reached.
To avoid reaching the max limit and let the subscription go into suspended mode, it is possible to create an alert when the cost reaches a percentage of the budget.
To configure a budget, go to “Subscriptions” or “Cost Management + Billing”, then click on Budgets and fill in the parameters for your budget:
Click next and set an alert:
Because you do not have only Synapse in your subscription, you may have VMs, SQL DBs, etc.
by Contributed | Jan 16, 2021 | Technology
This article is contributed. See the original author and article here.
(part 1 of my series of articles on security principles in Microsoft SQL Servers & Databases)
Principle of Least Privilege (POLP)
The first security principle, one that most System Administrators are familiar with, is the “principle of least privilege” (short: POLP). It demands that the required permissions for a task shall only grant access to the needed information or resources that a task requires. When permissions are granted, we shall grant the least privileges possible.
POLP is so crucial because initially it is the privileges that any attacker is targeting. When developing an application, using a least-privileged user account (LUA) is the first rule of engagement.
Note
User Account Control (UAC) in Windows is a feature that Microsoft developed to assist administrators in working with least-privileges by default and elevate to higher permission only when needed.
You may also know that Microsoft recommends separating service accounts. This security best practice is generally referred to as service account isolation and is related to POLP: Using distinct service accounts prevents increased privileges, which happens easily when you share an account to be used for multiple purposes and as a consequence, the privileges are merged. – This would violate the principle of least privilege.
Both POLP and service account isolation help reducing the attack surface (aka attack surface reduction).
– Read more on this topic here: SQL Server security – SQL Server | Microsoft Docs
and here: Surface Area Configuration – SQL Server | Microsoft Docs
Service account isolation also prevents between services if an attacker gained access to one service. You see how one thing is connected to another in Security?
Note
Lateral movement is a common attack strategy to move from one target to the next: If the main target (for example the database server) cannot be breached into directly, attackers try to gain foothold in some other server in the system within the same network and then launch other attacks to try to get to the final goal, server by server or service by service.
Principle of Least Privilege in the SQL realm
Let’s look at some examples within the SQL Server engine (applying to on-prem as well as our Azure SQL-products):
Example 1, read-access to data
A typical example within SQL Server would be: To allow a User to only read data from a small set of tables, ideally defined by a schema-boundary, we have the SELECT-Permission, grantable at the schema- (or even table-) level. There is no need to grant SELECT at the whole database, or to Grant anything other than SELECT.
In the code-snippet below we see that there are many tables in different schemas (Application, Purchasing, Sales) within the database WideWorldImporters. Instead of granting Select in the whole database, we chose to grant the user Shakti the permission at the schema scope. As long as this schema really contains only objects that Shakti needs access to, this is a best practice as it greatly reduces the management and reporting overhead compared to granting permissions at the object-level.
Tip
An alternative is to use stored procedures for all data access, which would allow even better control and completely hide the schema from Users.
That was easy, wasn’t it?
Example 2, creating user accounts
Unfortunately, not everything is implemented to always ensure POLP.
Let’s take another example:
You want to delegate the ability to create new Logins in SQL Server.
The minimal permission available is ALTER ANY LOGIN. Ok, so now this person can create new Logins, and maybe also Dropping them is ok.
But: With this permission comes the ability to change the password of any SQL Login (“ALTER LOGIN … WITH PASSWORD=’NewPassword’).
This can be an unwanted scenario as this would enable this person to essentially take over other accounts.
Note
This would not work if the Account were a Windows Domain or Azure Active Directory account. This is where a separation of the authenticating system from SQL Server has a real advantage. (This is a great example of Separation of Duties btw.)
Example 3, changing table structures
What about the following?
Say you want to allow a developer to add a set of new columns to the existing tables. (For example, for logging purposes, you need to include the timestamp of any new row.)
The minimal permission to change tables/add new columns is the ALTER-permission on each individual table (it cannot be done on schema-level).
In my example you can see that adding new columns works fine, also dropping the table is not allowed.
But:
Instead of adding new columns this user could also drop existing columns. This is covered under the same least permission/privilege:
Or: You want them to only create new tables but disallow to change existing tables. Since the required permissions for that are: CREATE TABLE on Database + ALTER on the schema, they could also drop tables. With permissions alone this cannot be solved. This is a common reason for the use of DDL Triggers as a preventative control. (I demonstrated an example of a DDL Trigger in this Blog-Article: Logging Schema-changes in a Database using DDL Trigger, which can easily be adjusted to prevent certain statements altogether by rolling them back.)
Conclusion
The more you dive into this subject and real-world implementation, the more you will realize that this apparently basic security principle is much tougher than you may initially have expected.
The permission system of SQL Server is very granular, vast, and continuously growing. (SQL Server 2019 provides 248 permissions and Azure SQL Database exposes 254 permissions as of December 2020.)
While some of the examples above are reasonable, we need to balance every decision for every new permission and look at it from multiple angles whenever a new functionality or command is implemented. For example:
- Which other commands and tasks are covered under the same permission?
- How do they relate to the functionality at hand?
- Is the use of the new functionality/command alone a common scenario?
Having permissions on parts of Table-structures, like adding columns but not dropping them, would increase the complexity of the permission-system and hence the compromise to have just one ALTER-permission on table DDL was made.
That said, I know there are examples where the balance is not right, and SQL Server can be improved, like TRUNCATE TABLE requiring ALTER on the table as well and others.
Feel free to let me know where you believe that POLP is seriously unbalanced and more granularity is required to reach compliance.
Happy securing
Andreas
Resources
by Contributed | Jan 16, 2021 | Technology
This article is contributed. See the original author and article here.
Introduction into security principles in the context of database systems
By Andreas Wolter
While many of us are practicing ‘social distancing’, and spend lots of time at home, I am finally finding the time to share some of the topics with the public that I have been working on since I joined Microsoft at the end of 2018.
In the recent years and with increasing frequency, one of the asks in terms of Security to the SQL Engine On-Prem as well as SQL Azure Database has been coming up with solutions to help accomplish “Separation of Duties”. This is a good thing, because it reassures my point of view that Separation of Duties is becoming increasingly important in IT and specifically Cloud-based systems.
On the other hand, we noticed that there is not a broad understanding in the technical community yet as to what Separation of Duties (aka SoD) really means and how it can be accomplished today. It occurs to me that the understanding is often vague and sometimes even contradicting depending on who you ask. It might therefore help to provide some context and guidance on what SoD really is and how it relates to other commonly referenced security principles that have been established over the last decades in IT.
If you are not already an expert in IT security, I hope that you will find this series useful.
Intro: Motivation
Security principles in information technology or cybersecurity (I won’t touch on physical security in these articles) exist as guidelines to assist design and decision processes in architecture, implementation, and reactive procedures when incidents happen. The purpose is to help designing for security in the first place, by using common proven patterns, and to be able to effectively assess a systems security.
Building secure systems from the start can be an expensive task, but over the years we have all seen security incidents which can cost millions and cause companies or even banks to go out of business. (i.e. see https://www.ibm.com/ae-en/security/data-breach )
One word of caution: Simply complying with these security principles provides no guarantee of preventing successful attacks. Some attackers invest a lot of time thinking to come up with ever new methods and exploit attack vectors which may not have been considered before.
But following these security principles helps to reach the following objectives:
- Reducing the blast radius of an attack
- i.e. attackers may not gain access to all targeted services because of partitioning or may not be able to elevate to all permissions to gain access to all documents
- Increasing the time for a successful attack
- this also goes back to #1 as it becomes harder to gain sufficient access
- Increasing the chances of early detection (!!)
- More controls and audits usually mean more chances of raising triggers or errors along the way
- Improving forensic abilities after detection
- Better audit trails allow for more successful investigations
Security a first
Therefore, I strongly advise to implement the proper security controls from start. And this is not just because it is common knowledge among IT architects that changing running systems is more expensive than making sure that Security is a main pillar in the architecture from the start.
To take it one step further: security should be THE FIRST pillar to be implemented. What I mean by that is that, ideally, nothing gets deployed before all security measures have been put in place. Otherwise, it is easily possible to admit backdoors or other security issues in the foundation, purposefully or not, that remain undetected. The very first measure therefore should be to put Auditing in place. We will talk more about Auditing in a later article.
Contents
This series of articles will provide an overview on the most commonly cited security principles and concepts which are often used when talking about Separation of Duties – or even intermingled with it – and briefly clarify their meaning and relation. Expect a lot of keywords (not buzzwords though, I promise)
Principle of Least Privilege (POLP) Need to know
Delegation of Authority Separation of Privilege
Audit Trail Separation of Duties
– These articles will be released one by one over the next weeks and the links will then be updated one by one as well.
One more principle you should keep in mind when designing security: “KISS” – Keep it simple, stupid
As already mentioned in my article from 2017 (Separation of Duties (SoD) and role-based security conception in SQL Server), it is absolutely vital to keep the user experience as simple as possible. Anything “too much” of an effort (and that can be just “too many clicks”) will lead to users to try to find ways around it. And they will.
Example
A common example of that is the shared Admin account. Instead of having one elevated account per person, often especially in small shops, developers share one common privileged account. Among other things this renders Auditing almost useless as no one can really tell who did what.
Note
Separation of concerns (SoC)
Over time I have heard it being used when actually “Separation of Duties” was meant. SoC is NOT a security principle and rather a basic programming design principle which leads to modular (or “functional”) programming. Hopefully, this clears up this common mix-up.
Wikipedia-Article: https://en.wikipedia.org/wiki/Separation_of_concerns
Let me know if you find this series helpful and what else you want to hear about in the future.
Andreas
Special thanks to
Raul Garcia, former SQL Security PM and “honorable member for life” :) – your knowledge in Security and SQL Security helped me make sure to not overlook anything important and meet a certain quality bar ;)
Steven Gott, one of our most senior Security Engineers, for your critical viewpoints which help me look ahead, although I know I can’t possible mention everything.
Ralf Dietrich from Sarpedon Quality Lab® Germany for countless hours of brainstorming about secure architectures even while being based in separate time zones.
“Security Logo” by pixabay is licensed under CC0
by Contributed | Jan 16, 2021 | Technology
This article is contributed. See the original author and article here.
Introduction into security principles in the context of database systems
By Andreas Wolter
While many of us are practicing ‘social distancing’, and spend lots of time at home, I am finally finding the time to share some of the topics with the public that I have been working on since I joined Microsoft at the end of 2018.
In the recent years and with increasing frequency, one of the asks in terms of Security to the SQL Engine On-Prem as well as SQL Azure Database has been coming up with solutions to help accomplish “Separation of Duties”. This is a good thing, because it reassures my point of view that Separation of Duties is becoming increasingly important in IT and specifically Cloud-based systems.
On the other hand, we noticed that there is not a broad understanding in the technical community yet as to what Separation of Duties (aka SoD) really means and how it can be accomplished today. It occurs to me that the understanding is often vague and sometimes even contradicting depending on who you ask. It might therefore help to provide some context and guidance on what SoD really is and how it relates to other commonly referenced security principles that have been established over the last decades in IT.
If you are not already an expert in IT security, I hope that you will find this series useful.
Intro: Motivation
Security principles in information technology or cybersecurity (I won’t touch on physical security in these articles) exist as guidelines to assist design and decision processes in architecture, implementation, and reactive procedures when incidents happen. The purpose is to help designing for security in the first place, by using common proven patterns, and to be able to effectively assess a systems security.
Building secure systems from the start can be an expensive task, but over the years we have all seen security incidents which can cost millions and cause companies or even banks to go out of business. (i.e. see https://www.ibm.com/ae-en/security/data-breach )
One word of caution: Simply complying with these security principles provides no guarantee of preventing successful attacks. Some attackers invest a lot of time thinking to come up with ever new methods and exploit attack vectors which may not have been considered before.
But following these security principles helps to reach the following objectives:
- Reducing the blast radius of an attack
- i.e. attackers may not gain access to all targeted services because of partitioning or may not be able to elevate to all permissions to gain access to all documents
- Increasing the time for a successful attack
- this also goes back to #1 as it becomes harder to gain sufficient access
- Increasing the chances of early detection (!!)
- More controls and audits usually mean more chances of raising triggers or errors along the way
- Improving forensic abilities after detection
- Better audit trails allow for more successful investigations
Security a first
Therefore, I strongly advise to implement the proper security controls from start. And this is not just because it is common knowledge among IT architects that changing running systems is more expensive than making sure that Security is a main pillar in the architecture from the start.
To take it one step further: security should be THE FIRST pillar to be implemented. What I mean by that is that, ideally, nothing gets deployed before all security measures have been put in place. Otherwise, it is easily possible to admit backdoors or other security issues in the foundation, purposefully or not, that remain undetected. The very first measure therefore should be to put Auditing in place. We will talk more about Auditing in a later article.
Contents
This series of articles will provide an overview on the most commonly cited security principles and concepts which are often used when talking about Separation of Duties – or even intermingled with it – and briefly clarify their meaning and relation. Expect a lot of keywords (not buzzwords though, I promise)
Principle of Least Privilege (POLP) Need to know
Delegation of Authority Separation of Privilege
Audit Trail Separation of Duties
– These articles will be released one by one over the next weeks and the links will then be updated one by one as well.
One more principle you should keep in mind when designing security: “KISS” – Keep it simple, stupid
As already mentioned in my article from 2017 (Separation of Duties (SoD) and role-based security conception in SQL Server), it is absolutely vital to keep the user experience as simple as possible. Anything “too much” of an effort (and that can be just “too many clicks”) will lead to users to try to find ways around it. And they will.
Example
A common example of that is the shared Admin account. Instead of having one elevated account per person, often especially in small shops, developers share one common privileged account. Among other things this renders Auditing almost useless as no one can really tell who did what.
Note
Separation of concerns (SoC)
Over time I have heard it being used when actually “Separation of Duties” was meant. SoC is NOT a security principle and rather a basic programming design principle which leads to modular (or “functional”) programming. Hopefully, this clears up this common mix-up.
Wikipedia-Article: https://en.wikipedia.org/wiki/Separation_of_concerns
Let me know if you find this series helpful and what else you want to hear about in the future.
Andreas
Special thanks to
Raul Garcia, former SQL Security PM and “honorable member for life” :) – your knowledge in Security and SQL Security helped me make sure to not overlook anything important and meet a certain quality bar ;)
Steven Gott, one of our most senior Security Engineers, for your critical viewpoints which help me look ahead, although I know I can’t possible mention everything.
Ralf Dietrich from Sarpedon Quality Lab® Germany for countless hours of brainstorming about secure architectures even while being based in separate time zones.
“Security Logo” by pixabay is licensed under CC0
by Contributed | Jan 16, 2021 | Technology
This article is contributed. See the original author and article here.
Paralyzed in an accident from the waist down, Arjun* thought he would never walk again.
However, the 30-year-old, backed by his loved ones and a generous community, soon turned to a team of technologists and scientists, including MVP Praveen Srivatsa, for a second chance at life. The result is an impressive feat: an exoskeleton powered by Microsoft technologies which restores mobility to those without it.
Praveen, a six-time MVP titleholder from Bangalore who served as the team CTO, says it was Arjun’s relatives who noted the potential of exoskeletons but their prohibitive price point. “Moreover, those exoskeletons could not help patients with an injury higher up the spinal cord due to lack of upper body strength,” he says. Not one to give up, the family rallied a team and raised more than $1 million to build a patented third-generation exoskeleton that could help people with any kind of injury to stand up and walk again.
The device is a robotic system powered by a NI cRIO controller and driven by a Windows Surface tablet. The software platform leverages Azure Web, Storage, Azure SQL and Azure Functions. The robotic device streams IoT data through the Azure IoT Hub and Stream Analytics makes it possible to monitor the robot behavior in real-time. The actual gait profile and customized rehabilitation are controlled by a proprietary machine learning algorithm that is being trained on Azure ML.
The device is currently in a premier hospital in Bangalore for patient rehabilitation and data collection for different types of injuries. “Our vision is to be able to provide highly personalized rehabilitation from different types of injuries and enable people from different walks of life to not only to get back their mobility, but also to be able to lead their own life with dignity,” Praveen says. “And we wish to do this in a very affordable manner so that it can impact the lives of a much wider base of people affected by different injuries.”
Praveen says the exoskeleton team enjoyed large community support from the early days of the project. Doctors started stepping in to provide guidance on how to rehabilitate patients with different injuries, and retired officials came forward to provide insights on how the government could support this initiative on a larger scale.
Moreover, institutes like IIIT Bangalore came forward to assist on the research around machine learning and gait algorithms. “In fact, the technology community stepped in and offered crowdsourced data collection for human gait to enable us to create a gait profile baseline for our algorithms,” Praveen says.
“This initiative could not have progressed at this pace without the strong support of the community around us,” he says.
Editor’s Note: Microsoft MVP has used an alias for the patient to protect their anonymity.
Recent Comments