European SharePoint Conference (ESPC20) – upcoming online training event

by Contributed | Oct 8, 2020 | Technology

This article is contributed. See the original author and article here.

Be Inspired by the power of Microsoft 365 at the European SharePoint Conference (ESPC20) Online – powered by Microsoft Teams.

You’ll find expert-led tutorials (workshops), visionary keynotes, and informative breakout sessions, delivered by the world’s best independent Microsoft 365 experts and the incredible Microsoft Redmond team behind the products.

Take time to learn, ask questions and engage with fellow delegates via 1:1 chats or group networking to discover top tricks and helpful insights – all to help plan and ready for the next phases of your productivity updates and new ways of working.

Book now

Jeff Teper (CVP, Microsoft) and other Microsoft leaders will be giving keynotes that spans the globe and topics, plus Microsoft speakers delivering breakout sessions among thought leaders and members of the community from around the world. Review all Microsoft keynotes and sessions below and start your world journey today.

What: European SharePoint Conference 2020 – Online to learn more. Session schedule lists times in CEST; recordings will be available to watch post conference for registered delegates.

When: October 14^th – 16^th, 2020 (keynotes & sessions)

October 12^th & 16^th (pre- and post-event tutorials (workshops))

Presenters: 100+ sessions, 120+ speakers (MVPs, RDs, Microsoft and community members)

Cost: €49 – €339 (session and/or tutorial; discounts available for groups) | Book now

Primary Twitter hashtag: #ESPC20 (join in) & follow @EuropeanSP

European SharePoint Conference 2020 – Online – October 14-16, 2020 (online training); pre- and post-event tutorial (workshops) October 12th & 16th.

ESPC20 Online offers you affordable, world-class Microsoft 365 learning at your fingertips, from wherever you are. Tune in live or catch up on-demand across 100+ sessions SharePoint, Office 365 & Azure experts including Microsoft team members, independent community leaders, RDs, MVPs & MCMs covering SharePoint, OneDrive, Yammer, Azure Arc, Power Apps, Microsoft Teams, Microsoft Stream, Microsoft Graph, Logic Apps, SharePoint Framework, Office 365 eDiscovery, PnP PowerShell, Microsoft Information Protection & Security, and so much more.

Hear from the best of the best – Microsoft Product Team members, RDs, MVPs, MCMs and independent experts.

All Microsoft keynotes, breakout sessions, and tutorial (workshop)

• Microsoft keynotes
  
  
  
  • Build Agile Business Processes with the Microsoft Power Platform by Charles Lamanna
  
  
  • Microsoft 365 Collaboration Update by Jeff Teper
  
  
  • Balancing Productivity and Employee Wellness with Microsoft 365 by Karuana Gatimu
  
  
  • Low Code and the Microsoft Cloud by Scott Guthrie
  
  
  
  


• Microsoft breakout sessions
  
  
  
  • ​Azure Hybrid – Learn about Hybrid Cloud Management with Azure by Thomas Maurer
  
  
  • Connecting to Microsoft Graph Services from Microsoft Teams by Bob German
  
  
  • Cortana – what’s new with your personal productivity assistant in Microsoft 365 by Malavika Rewari, A.J. Brush, Saurabh Choudhury, and Srikanth Sridhar
  
  
  • Customize productivity apps with lists and libraries, Power Apps, and Power Automate by Chakkaradeep (Chaks) Chinnakonda Chandran
  
  
  • Deployment and configuration of OneDrive sync by Gaia Carini and Carter Green
  
  
  • Developing solutions across Microsoft 365 with SharePoint Framework by Luca Bandinelli (Oct. 14^th)
  
  
  • Employee Engagement and Communities in Microsoft 365 by Murali Sitaram and Dan Holme
  
  
  • Engaging employees with Yammer and Microsoft 365 by Michael Holste
  
  
  • Expertise & Knowledge Networks: Microsoft’s Vision For a Successful Knowledge System In Your Organization by Naomi Moneypenny
  
  
  • Get started with Microsoft Lists by Lincoln DeMaris and Bharath Manoj Manda
  
  
  • Governance and management best practices for Microsoft 365 Groups by Vinay Jagannatha Rao
  
  
  • Integrating Robotics Process Automation (RPA) into Workflows by Fabian Williams
  
  
  • Kick-start your solution development with Microsoft 365 PnP by Vesa Juvonen
  
  
  • Microsoft 365 cloud data center strategy by Sean Emam and Sesha Mani
  
  
  • Monitor and manage SharePoint in Microsoft 365 by Dave Minasyan, Rk Menon, and Trent Green
  
  
  • Moving classic SharePoint search to Microsoft Search by Jyoti Pal and Mikael Svenson
  
  
  • Organize business information with Microsoft Lists by Miceile Barrett and Niket Jain
  
  
  • Personal productivity insights in Microsoft 365 – enhancing employee wellbeing by Malavika Rewari and Sumati Ahuja
  
  
  • Project Cortex: Knowledge discovery and content intelligence in Microsoft 365 by Naomi Moneypenny and Chris McNulty
  
  
  • Role-based access control in Microsoft 365 by Timothy Heeney
  
  
  • Search where you’re already working with Microsoft Search: Vision, Strategy, Roadmap by Robin Thomas and Kathrine Hammervold
  
  
  • SharePoint Communication Sites & the Modern Intranet: What’s New by Debjani Mitra and Andy Haon
  
  
  • The Intranet is Dead. Long Live the Intranet. The SharePoint Intranet Vision and Strategy by Adam Harmetz and Tejas Mehta
  
  
  • Transforming your Existing Classic SharePoint Sites into Modern Sites By Bert Jansen
  
  
  • Virtual health and collaboration with Microsoft Teams in healthcare by Evan Westenberger, Daniel Canning, Kristina Behr, and Tom McGuinness
  
  
  • What is the Microsoft Graph? by Jeremy Thake
  
  
  • What’s new in Security & Compliance in SharePoint and OneDrive by Sesha Mani
  
  
  • Working with files and lists in Microsoft Teams by Mark Kashman
  
  
  
  


• Microsoft tutorial (workshop) (pre- and post-event workshops)
  
  
  
  • Working with Microsoft Lists (Oct. 12^th) by Mark Kashman
  
  
  
  

Shout out to community members Tracy O’Connell, Ray Glasheen and the #ESPC20 crew who are working hard to make it a great, virtual event – supporting and promoting the knowledge and expertise that reaffirms this: Microsoft 365 has the best tech community in the world – one that spans all oceans and continents.

ESPC is a chance to escape your daily routine and take the time to focus on you and your professional development.

Go ESPC20, go!

– Mark Kashman, senior product manager | Microsoft 365 marketing 

Analyzing the Limits of Connection Scalability in Postgres

by Contributed | Oct 8, 2020 | Technology

This article is contributed. See the original author and article here.

One common challenge with Postgres for those of you who manage busy Postgres databases, and those of you who foresee being in that situation, is that Postgres does not handle large numbers of connections particularly well.

While it is possible to have a few thousand established connections without running into problems, there are some real and hard-to-avoid problems.

Since joining Microsoft last year in the Azure Database for PostgreSQL team—where I work on open source Postgres—I have spent a lot of time analyzing and addressing some of the issues with connection scalability in Postgres.

In this post I will explain why I think it is important to improve Postgres’ handling of large number of connections. Followed by an analysis of the different limiting aspects to connection scalability in Postgres.

In an upcoming post I will show the results of the work we’ve done to improve connection handling and snapshot scalability in Postgres—and go into detail about the identified issues and how we have addressed them in Postgres 14.

Why connection scalability in Postgres is important

In some cases problems around connection scalability are caused by unfamiliarity with Postgres, broken applications, or other issues in the same vein. And as I already mentioned, some applications can have a few thousand established connections without running into any problems.

A frequent counter-claim to requests to improve Postgres’ handling of large numbers of connection counts is that there is nothing to address. That the desire/need to handle large numbers of connection is misguided, caused by broken applications or similar. Often accompanied by references to the server only having a limited number of CPU cores.

There certainly are cases where the best approach is to avoid large numbers of connections, but there are—in my opinion—pretty clear reasons for needing larger number of connections in Postgres. Here are the main ones:

• 
  

  Central state and spikey load require large numbers of connections: It is common for a database to be the shared state for an application (leaving non-durable caching services aside). Given the cost of establishing a new database connection (TLS, latency, and Postgres costs, in that order) it is obvious that applications need to maintain pools of Postgres connections that are large enough to handle the inevitable minor spikes in incoming requests. Often there are many servers running [web-]application code using one centralized database.

  
  

   

  
  

  To some degree this issue can be addressed using Postgres connection poolers like PgBouncer or more recently Odyssey. To actually reduce the number of connections to the database server such poolers need to be used in transaction (or statement) pooling modes. However, doing so precludes the use of many useful database features like prepared statements, temporary tables, …

  
  

   

  
  


• 
  

  Latency and result processing times lead to idle connections: Network latency and application processing times will often result in individual database connections being idle the majority of the time, even when the applications are issuing database requests as fast as they can.

  
  

  
  Common OLTP database workloads, and especially web applications, are heavily biased towards reads. And with OLTP workloads, the majority of SQL queries are simple enough to be processed well below the network latency between application and database.
  
  

  
  

  Additionally the application needs to process the results of the database queries it sent. That often will involve substantial work (e.g. template processing, communication with cache servers, …).

  
  

  
  To drive this home, here is a simple experiment using pgbench (a simple benchmarking program that is part of Postgres). In a memory-resident, read-only pgbench workload (executed on my workstation ¹, 20/40 CPU cores/threads) I am comparing the achievable throughput across increasing client counts between a non-delayed pgbench and a pgbench with simulated delays. For the simulated delays, I used a 1ms network delay and a 1ms processing delay. The non-delayed pgbench peaks around 48 clients, the delayed run around 3000 connections. Even comparing on-machine TCP connections to a 10GBe between two physically close machines moves the peak from around 48 connections closer to 500 connections.
  
  

  
  


• 
  

  Scaling out to allow for higher connection counts can increase cost: Even in cases where the application’s workload can be distributed over a number of Postgres instances, the impact of latency combined with low maximum connection limits will often result in low utilization of the database servers, while exerting pressure to increase the number of database servers to handle the required number of connections. That can increase the operational costs substantially.
  
  

  
  

Surveying connection scalability issues

My goal in starting this project was to improve Postgres’ ability to handle substantially larger numbers of connections. To do that—to pick the right problem to solve— I first needed to understand which problems were most important, otherwise it would have been easy to end up with micro-optimizations without improving real-world workloads.

So my first software engineering task was to survey the different aspects of connection scalability limitations in Postgres, specifically:

1. Memory usage


2. Snapshot scalability


3. Connection model

By the end of this deep dive into the connection scalability limitations in Postgres, I hope you will understand why I concluded that snapshot scalability should be addressed first.

Memory usage

There are 3 main aspects to problems around memory usage of a large numbers of connections:

• Constant connection overhead, the amount of memory an established connection uses


• Cache bloat, the increase in memory usage due to large numbers of database objects


• Query memory usage, memory used by query execution itself

Constant connection overhead

Postgres, as many of you will know, uses a process-based connection model. When a new connection is established, Postgres’ supervisor process creates a dedicated process to handle that connection going forward. The use of a “full blown process” over the use of of threads has some advantages like increased isolation / robustness, but also some disadvantages.

One common complaint is that each connection uses too much memory. That is, at least partially, a common observation because it is surprisingly hard to measure the increase in memory usage by an additional connection.

In a recent post about measuring the memory overhead of a Postgres connection I show that it is surprisingly hard to accurately measure the memory overhead. And that in many workloads, and with the right configuration—most importantly, using huge_pages —the memory overhead of each connection is below 2 MiB.

Conclusion: connection memory overhead is acceptable

When each connection only has an overhead of a few MiB, it is quite possible to have thousands of established connections. It would obviously be good to use less memory, but memory is not the primary issue around connection scalability.

Cache bloat

Another important aspect of memory-related connection scalability issues can be that, over time, the memory usage of a connection increases, due to long-lived resources. This particularly is an issue in workloads that utilize long-lived connections combined with schema-based multi-tenancy.

Unless applications implement some form of connection <-> tenant association, each connection over time will access all relations for all tenants. That leads to Postgres’ internal catalog metadata caches growing beyond a reasonable size, as currently (as of version 13) Postgres does not prune its metadata caches of unchanging rarely-accessed contents.

Problem illustration

To demonstrate the issue of cache bloat, I created a simple test bed with 100k tables, with a few columns and single primary serial column index ². Takes a while to create.

With the recently added pg_backend_memory_contexts view it is not too difficult to see the aggregated memory usage of the various caches (although it would be nice to see more of the different types of caches broken out into their own memory contexts). See ³.

In a new Postgres connection, not much memory is used:

But after forcing all Postgres tables we just created to be accessed ⁴, this looks very different:

As the metadata cache for indexes is created in its own memory context, num_contexts for the “index info” contexts nicely shows that we accessed the 100k tables (and some system internal ones).

Conclusion: cache bloat is not the major issue at this moment

A common solution for the cache bloat issue is to drop “old” connections from the application connection pooler after a certain age. Many connection pooler libraries / web frameworks support that.

As there is a feasible workaround, and as cache bloat is only an issue for databases with a lot of objects, cache bloat is not the major issue at the moment (but worthy of improvement, obviously).

Query memory usage

The third aspect is that it is hard to limit memory used by queries. The work_mem setting does not control the memory used by a query as a whole, but only of individual parts of a query (e.g. sort, hash aggregation, hash join). That means that a query can end up requiring work_mem several times over ⁵.

That means that one has to be careful setting work_mem in workloads requiring a lot of connections. With larger work_mem settings, practically required for analytics workloads, one can’t reasonably use a huge number of concurrent connections and expect to never hit memory exhaustion related issues (i.e. errors or the OOM killer).

Luckily most workloads requiring a lot of connection don’t need a high work_mem setting, and it can be set on the user, database, connection, and transaction level.

Snapshot scalability

There are a lot of recommendations out there strongly recommending to not set max_connections for Postgres to a high value, as high values can cause problems. In fact, I’ve argued that myself many times.

But that is only half the truth.

Setting max_connections to a very high value alone only leads at best (worst?) to a very small slowdown in itself, and wastes some memory. E.g. on my workstation ¹ there is no measurable performance difference for a read-only pgbench between max_connections=100 and a value as extreme max_connections=100000 (for the same pgbench client count, 48 in this case). However the memory required for Postgres does increase measurable with such an extreme setting. With shared_buffers=16GB max_connections=100 uses 16804 MiB, max_connections=100000 uses 21463 MiB of shared memory. That is a large enough difference to potentially cause a slowdown indirectly (although most of that memory will never be used, therefore not allocated by the OS in common configurations).

The real issue is that currently Postgres does not scale well to having a large number of established connections, even if nearly all connections are idle.

To showcase this, I used two separate pgbench⁶ runs. One of them just establishes connections that are entirely idle (using a test file that just contains sleep 1s, causing a client-side sleep). Another to run a normal pgbench read-only workload.

This is far from reproducing the worst possible version of the issue, as normally the set of idle connections varies over time, which makes this issue considerably worse. This version is much easier to reproduce however.

This is a very useful scenario to test, because it allows us to isolate the cost of additional connections pretty well isolated. Especially when the count of active connections is low, the system CPU usage is quite low. If there is a slowdown when the number of idle connections increases, it is clearly related to the number of idle connections.

If we instead measured the throughput with a high number of active connections, it’d be harder to pinpoint whether e.g. the increase in context switches or lack of CPU cycles is to blame for slowdowns.

These results⁷ clearly show that the achievable throughput of active connections decreases significantly when the number of idle connections increases.

In reality “idle” connections are not entirely idle, but send queries at a lower rate. To simulate that I’ve used the the below to simulate clients only occasionally sending queries:

sleep 100ms

SELECT 1;

Cause

Together these results very clearly show that there is a significant issue handling large connection counts, even when CPU/memory are plentiful. The fact that a single active connection slows down by more than 2x due to concurrent idle connections points to a very clear issue.

A CPU profile quickly pinpoints the part of Postgres responsible:

Obviously the bottleneck is entirely in the GetSnapshotData() function. That function performs the bulk of the work necessary to provide readers with transaction isolation. GetSnapshotData() builds so called “snapshots” that describe which effects of concurrent transactions are visible to a transaction, and which are not. These snapshots are built very frequently (at least once per transaction, very commonly more often).

Even without knowing its implementation, it does make some intuitive sense (at least I think so, but I also know what it does) that such a task gets more expensive the more connections / transactions need to be handled.

Two blog posts by Brandur explain the mechanics and issues surrounding this in more detail:

• How Postgres Makes Transactions Atomic


• How to Manage Connections Efficiently in Postgres, or Any Database
  
  

Conclusion: Snapshot scalability is a significant limit

A large number of connections clearly reduce the efficiency of other connections, even when idle (which as explained above, is very common). Except for reducing the number of concurrent connections and issuing fewer queries, there is no real workaround for the snapshot scalability issue.

Connection model & context switches

As mentioned above, Postgres uses a one-process-per-connection model. That works well in a lot of cases, but is a limiting factor for dealing with 10s to 100s of thousands of connections.

Whenever a query is received by a backend process, the kernel needs to perform a context switch to that process. That is not cheap. But more importantly, once the result for the query has been computed, the backend will commonly be idle for a while—the query result has to traverse the network, be received and processed by the application, before the application sends a new query. That means on a busy server another process/backend/connection will need to be scheduled—another context switch (cross-process context switches are more expensive than doing process-kernel-same process, e.g. as part of a syscall).

Note that switching to a one-thread-per-connection model does not address this issue to a meaningful degree: while some of the context switches may get cheaper, context switches still are the major limit. There are reasons to consider switching to threads, but connection scalability itself is not a major one (without additional architectural changes, some of which may be easier using threads).

To handle huge numbers of connections a different type of connection model is needed. Instead of using a process/thread-per-connection model, a fixed/limited number of processes/threads need to handle all connections. By waiting for incoming queries on many connections at once and then processing many queries without being interrupted by the OS CPU scheduler, efficiency can very significantly be improved.

This is not a brilliant insight by me. Architectures like this are in wide use, and have widely been discussed. See e.g. the C10k problem, coined in 1999.

Besides avoiding context switches, there are many other performance benefits that can be gained. E.g. on higher core count machines, a lot of performance can be gained by increasing locality of shared memory, e.g. by binding specific processes/threads and regions of memory to specific CPU cores.

However, changing Postgres to support a different kind of connection model like this is a huge undertaking. That does not just require carefully separating many dependencies between processes and connections, but also user-land scheduling between different queries, support for asynchronous IO, likely a different query execution model (to avoid needing a separate stack for each query), and much more.

Conclusion: Start by improving snapshot scalability in Postgres

In my opinion, the memory usage issues are not as severe as the other issues discussed. Partially because the memory overhead of connections is less big than it initially appears, and partially because issues like Postgres’ caches using too much memory can be worked around reasonably.

We could, and should, make improvements around memory usage in Postgres, and there are several low enough hanging fruits. But I don’t think, as things currently are, that improving memory usage would, on its own, change the picture around connection scalability, at least not on a fundamental level.

In contrast, there is no good way to work around the snapshot scalability issues. Reducing the number of established connections significantly is often not feasible, as explained above. There aren’t really any other workarounds.

Additionally, as the snapshot scalability issue is very localized, it is quite feasible to tackle it. There are no fundamental paradigm shifts necessary.

Lastly, there is the aspect of wanting to handle many tens of thousands of connections, likely by entirely switching the connection model. As outlined, that is a huge project / fundamental paradigm shift. That doesn’t mean it should not be tackled, obviously.

Addressing the snapshot scalability issue first thus seems worthwhile, promising significant benefits on its own.

But there’s also a more fundamental reason for tackling snapshot scalability first: While e.g. addressing some memory usage issues at the same time,qs switching the connection model would not at all address the snapshot issue. We would obviously still need to provide isolation between the connections, even if a connection wouldn’t have a dedicated process anymore.

Hopefully now you understand why I chose to focus on Postgres snapshot scalability first. More about that in my next blog post.

1. 
   

   2x xeon gold 5215, 192GiB of RAM, kernel 5.8.5, debian Sid ↩︎
   
   

   
   


2. 
   

   creating 100k tables with psql:

   
   
   
   

   postgres[2627319][1]=# SELECT format(‘begin;create table foo_%1s(id serial primary key, data1 int, data2 text, data3 json);commit;’, g.i) FROM generate_series(1, 100000) g(i)
   
   gexec
   
   COMMIT
   
   COMMIT
   
   …
   
   

   
   
   
   ↩︎
   
   


3. 
   

   query cache memory usage:

   
   
   
   

   WITH RECURSIVE contexts AS (
   
       SELECT * FROM pg_backend_memory_contexts),
   
       caches AS (
   
           SELECT *
   
           FROM contexts
   
           WHERE name = ‘CacheMemoryContext’
   
         UNION ALL
   
           SELECT contexts.*
   
           FROM caches
   
           JOIN contexts ON (contexts.parent = caches.name)
   
       )
   
   SELECT
   
       name, parent,
   
       sum(total_bytes) size_bytes,
   
       pg_size_pretty(sum(total_bytes)) size_human,
   
       count(*) AS num_contexts
   
   FROM caches
   
   GROUP BY name, parent
   
   ORDER BY SUM(total_bytes) DESC;
   
   

   
   
   
   ↩︎
   
   


4. 
   

   Query to access all tables named foo*:

   
   
   
   

   DO $$
   
     DECLARE
   
       cnt int := 0;
   
       v record;
   
     BEGIN
   
       FOR v IN SELECT * FROM pg_class WHERE relkind = ‘r’ and relname LIKE ‘foo%’ LOOP
   
           EXECUTE format(‘SELECT count(*) FROM %s’, v.oid::regclass::text);
   
           cnt = cnt + 1;
   
           IF cnt % 100 = 0 THEN
   
               COMMIT;
   
           END IF;
   
       END LOOP;
   
       RAISE NOTICE ‘tables %1’, cnt;
   
     END;$$;
   
   

   
   
   
   ↩︎
   
   


5. 
   

   Even worse, there can also be several queries in progress at the same time, e.g. due to the use of cursors. It is however not common to concurrently use many cursors. ↩︎
   
   

   
   


6. 
   

   This is with pgbench modified to wait until all connections are established. Without that pgbench modification, sometimes a subset of clients may not be able to connect, particularly before the fixes described in this article. See this mailing listpost for details. ↩︎
   
   

   
   


7. 
   

   Idle Connections vs Active Connections:

   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   

   PG Version	Idle Connections	Active Connections	TPS
   12	0	1	33457
   12	100	1	33705
   12	1000	1	30558
   12	2500	1	26075
   12	5000	1	23284
   12	10000	1	14496
   12	0	48	1032435
   12	100	48	960847
   12	1000	48	902109
   12	2500	48	759723
   12	5000	48	702680
   12	10000	48	521558

   
   ↩︎


8. 
   

   Mostly Idle Connections vs Active Connections:

   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   

   PG Version	Less active Connections	Active Connections	TPS
   12	0	1	33773
   12	100	1	29074
   12	1000	1	25327
   12	2500	1	19752
   12	5000	1	9807
   12	10000	1	6049
   12	0	48	1040616
   12	100	48	953755
   12	1000	48	759366
   12	2500	48	733000
   12	5000	48	636057
   12	10000	48	416819

   
   ↩︎

Conditional Access Office 365 Suite now in GA!

by Contributed | Oct 8, 2020 | Technology

This article is contributed. See the original author and article here.

Howdy folks,

Today we’re announcing GA of Conditional Access for the Office 365 Suite! This makes it a whole lot easier to configure Conditional Access policy for Office 365. With a single click, you confidently set policy on all of the Office 365 apps, including Exchange Online, SharePoint Online, and Microsoft Teams, as well as micro-services used by these well-known apps.

We released Conditional Access for the Office 365 Suite to public preview in February, right before many employees switched to remote work in response to COVID-19, causing organizations to quickly update their access policies. The simplified experience helped thousands of organizations make this shift, so we also added it to our secure remote access guidance. Since February, we’ve seen the usage of this policy grow to over 7 million active users.

With the GA release, if you haven’t already updated your policies to use the Office 365 Suite, you should do so now.  Review any policies you have targeting individual Office apps to see if they can switch to Office 365 policy targeting.

Using the Office 365 Suite ensures your users will have consistent policy requirements across Office 365. Office 365 apps use shared services for an integrated experience, like access to files, calendar, and contact information. Inconsistent policy across these services result in end-users being interrupted or blocked at unexpected times with additional security prompts.

Targeting the Office 365 Suite also means you don’t need to watch for new apps and manually update your policy as they become available—Conditional Access does that for you.

Share your feedback!

The Office 365 Suite in Conditional Access is the best way to apply policy to Office 365 apps. Here is some additional documentation so you can learn more about the Conditional Access for the Office 365 suite.

Thank you to all our users who tried the preview and provided feedback. GA of this feature is just a start:  we’re continuing to build richer options for Conditional Access and Office 365 so please keep the feedback coming.

Best Regards,

Alex Simons (Twitter: @alex_a_simons)

Corporate Vice President Program Management

Microsoft Corporation

How to operationalize Enterprise-Scale with Infrastructure-as-Code via AzOps

by Contributed | Oct 8, 2020 | Technology

This article is contributed. See the original author and article here.

In my last blog I talked about the policy-driven governance with Enterprise-Scale. In this fourth article of my little series about Enterprise-Scale, I will write about one of the options to operationalize  Enterprise-Scale: AzOps.

Problem statement

Using Infrastructure-as-Code (IaC) to deploy and manage complex platform infrastructure deployments can quickly become quite tricky. For example, existing IaC code(s) to deploy an Azure platform infrastructure (management groups, policies, RBAC roles, etc.) must be updated when a new Azure resource is added to the platform infrastructure, and available CI/CD pipelines maybe have to be updated too, in order to address new deployment parameters. If the platform has to be changed, for example an additional management group hierarchy is introduced, may also lead to significant coding effort (adding smart logic), with all associated additional work. Furthermore, even more complexity is added when multiple platform engineers are supposed to work on the same IaC for the platform. In the end, it may be that one engineer is responsible for the entire deployment, to avoid breaking all implementation details. And finally, using IaC usually relies on the principle that the source code is the immutable source of truth. As a consequence of this, the expectation is that no changes should be made via other channels, such as the Azure portal or Azure CLI.

A possible solution – AzOps

AzOps is a reference implementation and represents a possible solution to the mentioned challenges with many IaC implementations. It’s important to call out that AzOps is an optional component to operationalize Enterprise-Scale, and there are other options as well. AzOps is orthogonal and can be used in any Azure environment, to transition to IaC once an organization is ready for it. Therefore, AzOps exists in a dedicated GitHub repository.[1]

What is AzOps?

AzOps is an atomic, primitive CI/CD pipeline for the Azure platform scopes (tenant, management groups, and subscriptions). AzOps allows organizations to bring their own Azure Resource Manager (ARM) templates, and deploy them at the designated scope. It helps to keep the Azure platform up-to-date with the ARM-based configuration stored in a Git repository, but it can also be used to update the Git repository with changes made via alternate methods and channels, e.g. via the Azure portal. AzOps is packaged as Docker-compatible container image and invoked as part of an GitHub Action or Azure DevOps Pipelines (or another CI/CD tool). AzOps is used to deploy ARM templates at the scopes where the service principal (used Azure Active Directory identity) has been grated access.

Now a concrete example related to policies:

• An engineer works on a new policy (policyDefinitions), and stores it in the Git repository (the new policy is nothing more than an ARM template).


• The new policy (ARM template) will used by AzOps to deploy it (Microsoft.Authorization/policyDefinitions resource) at the designated scope.

The engineer can concentrate solely on the policy development (one ARM template), no other changes are required with regards to IaC or CI/CD pipeline.

Azure platform infrastructure in a Git repository

As mentioned above, AzOps enables to store the ARM-based configuration (Azure platform infrastructure) in a Git repository. This includes the Azure resources managementGroups, subscriptions, policyDefinitions, policySetDefinitions, policyAssignments, roleDefinitions, and roleAssignments.[2]

The ARM-based configuration stored in a Git repository may look as follows:

Figure 1: Azure configuration hierarchy stored in a Git repository.

What you can see in figure 1 is the following:

I personally really like the approach with AzOps, as it is based on native tools such as Azure PowerShell and ARM templates, and it has been continuously developed and adapted to cover new requirements. Furthermore, the implementation option with AzOps is also described in Enterprise-Scale, in the chapter “[Optional] Getting started with Infrastructure-as-code”.[3] 

Finally, another big thank you to @KristianNese for reviewing and providing feedback.

[1] https://github.com/Azure/AzOps

[2] https://github.com/Azure/AzOps#discovery

[3] https://github.com/Azure/Enterprise-Scale/blob/main/docs/Deploy/getting-started.md

What’s New: HTML and Markdown support for incident comment

by Contributed | Oct 8, 2020 | Technology

This article is contributed. See the original author and article here.

This installment is part of a broader series to keep you up to date with the latest features in Azure Sentinel. The installments will be bite-sized to enable you to easily digest the new content.

The process of investigation is partly a science and partly an art. The analyst is ultimately building a storyline of what happened to determine whether the chain of events is the result of a malicious actor (often attempting to mask their actions/nature), a normal business/technical process, an innocent mistake, or something else.

As the analyst builds this understanding, they must capture a complete record with their conclusions and reasoning/evidence for future use (case reviews, analyst self-education, re-opening cases that are later linked to active attacks, etc.). Documentation is imperative to the lifecycle of an investigation and having the ability to support various ways to document and add context is necessary for SOC analysts.

With that in mind, we are delighted to announce that we have added the support for rendering HTML and Markdown in the Sentinel incident’s comment section! This gives you the ability to provide immediate value for documenting with additional context and the capability to view the context in a more meaningful way. Learn more about markdown using this reference and the cheatsheet.

For eaxmple, it’s very easy to make some words **bold** and other words *italic* with Markdown. You can even [link to evidence!](http://google.com?q=evidence). The result would look like this:

Additionally, we increased the allowed size of comments from 1000 chars to 3000 chars. This will ensure that you have enough room to include contextual information to the comment field in Azure Sentinel.

Note: At the moment, we do not have an HTML/Markdown visual editor in the comments section. However, you can input HTML/Markdown directly through the UI using the Markdown syntax, or send it using a Logic App or the API and the comment section will render it. We plan to add an HTML/Markdown editor to the comments section. Additionally, we also plan to support editing and deleting of comments.

How to:

Use case: Formatted enrichments using playbooks

Incident enrichment is a major use case is the process of incident triage, investigation, and response. This feature can also be used in tandem with playbooks (LogicApps) to enrich incidents with external information.

The example below shows a comment added by an IP enrichment playbook that captures information from VirusTotal and adds it as a formatted text to the incident comment section:

Get started today!

We encourage you to utilize this new set of features to enhance the way that you are documenting the investigation process and adding contextual information to incidents. 

Try it out and let us know what you think!

Microsoft and Arm Partner to Deliver an End-to-End Intelligent Edge Device Toolchain

by Contributed | Oct 8, 2020 | Technology

This article is contributed. See the original author and article here.

Microsoft is in the business of making digital transformation easy, fast, and cost effective for our customers. As we look at the opportunity presented by the rise of AI, it is essential to recognize silicon as a foundational component to the success of all intelligent edge devices, and the services and software that light them up. At the center of our silicon strategy is a commitment to support and partner with a broad and diverse ecosystem of silicon partners, device OEMs, and end customers.

TAM estimates for AI inferencing silicon at the edge are as high as $51.6 billion by 2025 with 70+ specialty AI companies working on a wide variety of chip-related technologies. Silicon diversity in a device ecosystem leads to innovation, heathy competition, and a choice of solutions across the value chain for developers, device makers, and end customers. However, Microsoft AI developer surveys indicate that the #1 reason developers do not try new silicon options is the cost of learning a new toolchain. Developers tell us that it can take more than a year for new silicon to be properly supported with the right optimizations in toolchains. In addition, we hear that many of the available toolchains lack a robust end-to-end experience, requiring developers to stitch together solutions, experimenting along the way. To accelerate growth for this fast-emerging market, vertically-integrated E2E solutions that lower time-to-value for our partners and customers are necessary.

I am delighted to announce that Microsoft and Arm are collaborating to deploy an integrated Azure-based toolchain targeting Arm-based silicon and delivering a unified and streamlined AI experience for developers. Microsoft and Arm have been close partners for many years. This deepened partnership will enable developers to easily target a broad range of Arm-based silicon in AI-enabled devices like intelligent computer-vision-enabled cameras, connected vehicles, AI gateways, and intelligent appliances.

Arm’s leading silicon architecture combined with their broad ecosystem and commitment to optimizing developer experiences will enable this partnership to accelerate AI innovation for IoT devices, delivering better solutions and a better future for end-users. The new wave of AI devices are much closer to where the data is generated, typically near the people working in factories, warehouses, stores, office buildings, and hospitals. These devices need to accommodate that proximity to people—no loud cooling fans, and inconspicuous form factors that are easier to use. Arm-based silicon can enable SoCs (system on chips) that consume little power, generate minimal heat, and can even eliminate the need for loud cooling fans. This is ideal for edge AI devices.

Edge AI workloads and devices need specialized silicon. We see that the adoption of AI will span multiple time-horizons realized in three progressive stages: Connected, Intelligent, and Autonomous. With more devices connected to the cloud, the demand for dashboards to manage and monitor these devices remotely has become a normal course of business. As we progress further into the intelligent horizon, being ready for devices that provide a flood of insights can provide enterprises with significant economic and competitive advantages. These Intelligent devices will disrupt value chains as they become the sensors, monitors and communicators. The autonomous horizon is almost upon us and will be the most disruptive with data-driven workflows executed between machines with little to no human interaction required. Companies slow to adopt this technology may fall so far behind that it becomes economically impossible to catch up.

Microsoft will collaborate with our ecosystem partners to plan across these multiple horizons, focused on customer-driven scenarios that inform both Intelligent Edge device portfolios and the silicon requirements to light them up. Each horizon will include a set of fundamental device capabilities, but as use cases evolve across various industry segments, we will need a broader and more comprehensive view of silicon requirements and coverage to succeed.

In addition to our focus on silicon diversity for intelligent Azure-connected devices, we are also working to address two significant challenges we hear from others in the ecosystem: finding the right devices to meet a customer’s needs, and finding the right audience for their devices and solutions. With over 30 billion active IoT devices in the world with 400 percent growth over the past three years, the industry is moving incredibly fast. Connecting the right audience with the right product will only become more difficult. Our recent announcement of the Azure Certified Device program is intended to help with these challenges. You can find more details at Getting Started with the Azure Certified Device program.

At the heart of this device certification program is a promise to device builders that they can not only get their devices to market quickly, but also more easily differentiate and promote their devices. It’s also a promise to buyers that they can easily find devices that meet their needs, and purchase those devices with confidence that they meet Microsoft standards for certification and are ready to work with Azure.

To support these promises, Microsoft is working closely with the silicon ecosystem to launch an Azure- Ready Silicon program. This program is a forward-looking effort to inform future silicon requirements for the device certification program, ensuring device builders have diverse and differentiated silicon that easily runs on Azure and is readily available. This involves aligning on fundamental silicon capabilities including security, connectivity, and compute—juxtaposed against a spectrum of real customer scenarios, requirements and device categories.

We need to collectively execute against a long-term view to intercept the future requirements for the Intelligent Edge. We know that the best things happen at-scale when people have choice. We are committed to helping foster innovation and diversity within the broader silicon and device ecosystems, and making it easy, fast, and cost-effective for our customers. Our collaboration with Arm to facilitate an end-to-end AI toolchain to simplify development is an important part of this strategy, and we’re very excited to continue tackling these AI and Intelligent Edge challenges together with them.