MCM: Core Active Directory Internals

MCM: Core Active Directory Internals

by | Oct 15, 2020 | Technology

This article is contributed. See the original author and article here.

 


First published on TechNet on Jul 22, 2012


 


Disclaimer: For brevity and to get some key points across, quite a bit of detail about about Active Directory, the underlying database, and replication have been purposely ommitted from this blog. 


 


Now, there is no possible way to cover every possible detail from every day during the MCM. Consequently, my plan is to cover the concepts and topics that are most important. Before jumping into topics, I want to set the scene for you.


 


I flew up to Seattle on Super Bowl Sunday back in February 2012 to our Redmond, WA headquarters. When I showed up at 9 AM, I was greeted by a classroom full of students. These students were from various parts of the world that had flown in to take this class. Some of them were from Microsoft including PFE or MCS while some were external. It was immediately evident that these students were seasoned professionals having anywhere from 7-20 years of experience within IT and having had experience with Active Directory since the very beginning. One thing that I did like about being amongst professionals of this level is that there were very few technical pissing matches because everyone knew someone in the classroom was probably much smarter than them.


 


Secondly, I want to stress the atmosphere of the classroom and materials. When the instructors were presenting the materials, it was pretty much expected they you partially knew what they’re talking about. The slides are pretty minimal on details. This class delivers the goods by:  [Filling in on details about AD through presentations] + [Classroom discussion] + [Labs] + [Self-Study] + [Group Study]. This is one of the reasons that the MCM is such a great experience because so much of it involves the class working together as a whole or within smaller groups. By going through the class, you begin to forge good relationships and bonds with people in the class. It’s as much an exercise in professional networking as it is learning.


 


Also, I cannot stress this point enough. If you take away one thing from this blog, let it be this: This class is not so you memorize every little detail about Active Directory, or what we call “Geek Trivia”. For example, can you recite from memory, the schema attribute value that enables containerized indexing? The exams or labs will never test you on this sort of thing but you will be expected to know what containerized indexing is, where to set it, and then through your own research, you can figure out what value needs to be set.


 


So, let’s begin. First off, I want to acknowledge Chris Davis, a PFE from Microsoft. His detailed notes helped ensure that I didn’t miss any important details. The first day was Core Active Directory internals.


 


Some of the things we covered were the following:



  1. What really is Active Directory?

  2. What’s inside the Active Directory database?

  3. Let’s see some AD internals.

  4. What really is a Global Catalog server?

  5. What are linked values?

  6. What are Phantom Objects?

  7. What really is the Infrastructure Master FSMO role?


 


What really is Active Directory?


 


Had you asked me what Active Directory was before I went to the Masters class, I probably would have just answered, “An LDAP-enabled database with many dependent LDAP-enabled applications and services sitting on top of it including Kerberos, Authentication, DNS, etc.” Now, having gone through the Master class, my answer would change to “A distributed Jet/ESE database that’s exposed through LDAP by the Directory System Agent (DSA) with many dependent LDAP-enabled applications and services sitting on top of it including Kerberos, Authentication, DNS, etc.”


 


Now, that I just explained what AD is at its lowest level, a Jet database, why in the world would Microsoft choose Jet over say, a SQL database? SQL is so very well known, easy to access and manipulate; it almost sounds like a match made in heaven.  Jet was chosen because it’s a ridiculously simple and fast database. If Active Directory was going to be the center of many enterprises, it had to be fast and Jet delivers on that promise in spades.


 


BrandonWilson_0-1602792216323.png


 


I like to describe the Directory System Agent (DSA) as the man behind the curtain, the bouncer, and the translator. It’s the component that talks to the database but also enables LDAP. Sorry to break it to you, but at the database level, distinguished names like ‘CN=users,DC=Contoso,DC=local’ don’t exist. It’s the DSA that creates this LDAP path based on the data in the underlying Jet database; this will make more sense in the next section. It also enforces data integrity, which data types are allowed for certain attributes. It really is the magic that creates this awesome LDAP database we call Active Directory. Jet makes it fast, the DSA makes it LDAP.


 


Now, within the ntds.dit file, there are actually many tables of data. The tables that are of most interest to us are the data table, which contains all the users, groups, OU’s. The link table, which contains any linked attributes for example, the members of a group. And lastly the SD table, which contains security descriptors or permissions that are assigned throughout Active Directory.


 


BrandonWilson_1-1602792216328.jpeg


Structure of NTDS.dit


 


Let’s first take a look at data table. One easy way to do this without some fancy third party tools is to run LDP.exe and leverage an operational attribute called ‘DumpDatabase’. Do note that this forest is called contoso.local with a child domain named child.contoso.local.


 


Start Ldp.exe on the domain controller.



  1. Connect locally, and then bind as an Domain Admin.

  2. Click Modify on the Browse menu.

  3. Edit for Attribute: dumpdatabase.

  4. Edit for Values: name ncname objectclass objectguid instancetype. You must leave one space between the attributes.

  5. Click Enter. The Entry List box contains the following entry:[Add]dumpdatabase:name ncname objectclass objectguid instancetype

  6. Click the Extended and Run options.

  7. The %systemroot%NTDSNtds.dmp file is created, or you receive an error message in Ldp.exe that you must investigate.


Source: http://support.microsoft.com/kb/315098


 


Data Table


 


The file created, ntds.dmp, is a text file that can be opened in notepad although the file size will depend on how big your Active Directory database is and we all know that notepad doesn’t like huge files. 


BrandonWilson_2-1602792216330.png


 


Nonetheless, once you open it in notepad, what you’re looking at is the data table from Active Directory and it should look something like this:


 


Disclaimer: I excluded some columns from this picture that wouldn’t fit nor was relevant to this blog.


 


BrandonWilson_3-1602792216343.jpeg


 


Here is a key for some of the above terms:


 


DNT: Distinguished Name Tag. Essentially is a primary key to identify each row within the database.


PDNT: Parent Distinguished Name Tag. Indicates which object in the database is the parent object of this object. References another objects DNT.


NCDNT: Naming Context Distinguished Name Tag. Indicates which “partition” this object belongs to. References the root of a partition’s DNT.


 


The first thing you’ll notice is that all the partitions in Active Directory are represented in this one data table. This is why we call them logical partitions. So, how does Active Directory keep track of the different partitions and which objects belong to which partitions? This is where the DNT, PDNT, NCDNT values you see above come into play. The PDNT value tells each object what their parent object is plus the NCDNT value tells the object which partition it belongs to.


 


BrandonWilson_4-1602792216348.jpeg


 


In the above diagram, you’ll notice that the DNT is just like a unique identifier where each row as a different value. The PDNT on each object tells us which object within the data table is its parent object. Additionally, you’ll notice the NCDNT on the Dave user account tells us that he belongs in the contoso.local domain partition. You’ll notice that the users container also has a NCDNT of 1788. This just indicates that the users container also belongs to the contoso.local domain partition. NCDNT tells us which partition each object belongs to.


 


The DSA then uses this information to map out the hierarchy of all objects and their partitions and delivers them in LDAP syntax. When I realized that almost all data and partitions in Active Directory are in this one data table and just organized by these hierarchal numbers, it forever changed my understanding of Active Directory. You’ll fully understand what I mean in a little bit.


 


Now, let’s also take a look at a GC at this low level. The official definition of a GC is that it contains a partial attribute set of every object in the Active Directory forest. While that is true, once again, all of this is stored in the one data table in Active Directory and organized by DNT’s, PDNT’s and NCDNT’s:


 


BrandonWilson_5-1602792216357.jpeg


 


The diagram above is a dump from a forest-root GC. Once again, you’ll notice the PDNT references the parent object. The NCDNT references what partition this object belongs to. And the PDNT on the child object, which is the root of the child domain, points to the DNT of contoso.local. We know this is a GC because these objects here at the bottom are from the child domain, which only a GC would have.


Key Takeaway: Active Directory does not have different tables to store the different partitions including the GC partition. Everything is stored in the one data table which is logically and hierarchy organized.


 


Now that I knew and understood Active Directory in this way, my mind started to open up and understand things that I couldn’t fully comprehend before.


 


Link Table & Linked Values


 


Linked values are a way of telling Active Directory that two attributes are related to one another. For example, on groups, we have an attribute called member that contains all the users that belong to that group. On each user account, we have an attribute called memberof that will show you all the groups that that user belongs to. Consequently, the member and memberof attributes are linked values that tell Active Directory they are related. Earlier, I mentioned the link table in the Active Directory database. It contains all the information about these linked values and in this case, who’s a member of these groups. Do remember that the link table may also contain information about other linked values as well, like the attributes ‘DirectReports’ and ‘ManagedBy’. Here is an example of the Dave user account belonging to the administrators group in contoso.local:


 


BrandonWilson_6-1602792216361.jpeg


 


So when you go to the properties of the administrators group to see who is a member, the database would take the administrators DNT of 3566, search the link table for all matching link_DNT values, and then return backlink_DNT values, which would correspond to a user or group within the DB that are members of that group.


 


In the reverse, when I go to the properties of the Dave account to see what groups he belong to, the database takes my DNT of 3830 and searches the link table for all matching backlink_DNT values, and then returns the link_DNT values, which would correspond to groups within the DB that I belong to.


 


Key Takeaway: Anything that is linked, like member and memberof attributes, must reference a physical object in the database. This is for purposes of referential integrity and it must have a corresponding DNT value, which means it will have its own row in the database. Contrast this with any generic multi-valued attribute within AD. If it isn’t linked, you can go ahead and add any value you want to it.


 


With that being said, let’s say that I log onto the child domain (child.contoso.local) and want to make the user account Dave, from the forest root, an administrator in the child domain. Now remember that this child DC is NOT a GC so he wouldn’t have a copy of the Dave user account in his data table. Also, remember that when you add someone to a group, they MUST physically exist in the local data table in Active Directory.


 


Does that mean that I have to make this child DC a GC so Dave would exist in the data table so we could then then add him to the administrators group?


 


Phantom Objects


 


Now, what actually happens under the hood is the DC creates what’s called a phantom object in the data table that references the Dave account in the forest root. This phantom object is now a real object with its own DNT and exists in the data table in the child domain on all non-GC’s. Now, he can properly be added to the administrators group. Let’s take a look at this under the hood from the child DC that is not a GC:


 


BrandonWilson_7-1602792216366.jpeg


 


The first clue that this is a phantom object is because OBJ=False. But if we compare this phantom object to the actual user account in the forest-root domain, it looks like this:


 


BrandonWilson_8-1602792216374.jpeg


 


Since this child DC isn’t a GC and didn’t have a copy of forest-root Dave account but had to still add Dave to the administrators group, it has to create a representation of Dave in its local database because the rules of linking state that the object must exist in the local data table and have a valid DNT.


 


Key Takeaway: Remember that GC’s don’t have nor need phantom objects because they have a row in their data table for every object in the forest so phantoms objects aren’t necessary. Non-GC’s only have the objects from their local domain so they have to create phantom objects to represent accounts from other domains.


 


Now, let’s take a look at the link table on this DC in the child domain from adding the Dave account in the forest root to the administrators group in the child domain:


 


BrandonWilson_9-1602792216385.jpeg


 


Tying It all Together


 


Now, why does any of this matter? Well, do you remember that recommendation that Microsoft made a long time ago about not putting the Infrastructure Master on a Global Catalog server? Everything I explained above is why. Let’s step through it one more time to make it clear. Before we do, let’s summarize some absolutes about Active Directory:



  1. Every domain controller is personally responsible for maintaining their own data table and how that data is internally linked. Internally, the DB on each DC may not be identical but the outcome will be the same.

  2. On each DC, to add a user to a group, that user must physically be present in the local data table either as a user account or a phantom object.

  3. A Global Catalog Server has a partial copy of every object in the forest in its data table. Objects from other domains don’t have all their attributes populated but nonetheless are present. Because of this, it doesn’t need phantom objects because it has the real objects locally.

  4. A Domain Controller that isn’t a GC doesn’t have a copy of every object in the forest in its data table. It only contains objects from its own domain. Because of this, it has to create phantom objects to reference the real objects from other domains.

  5. The infrastructure master is responsible for updating or deleting phantom objects if/when they change. For example, does the actual Dave account in the forest root still exist? Has he been moved or renamed? This process runs every 2 days and asks this question and then either updates or deletes the phantom objects accordingly.


 


One day, the forest-root Dave account gets deleted. The infrastructure Master role is running in the child domain on a global catalog server. Let’s go through it step-by-step:


 


Disclaimer: AD replication occurs at a much higher level than this and does not occur based on DNT values. I am just doing it this way to put it into the context of this blog. Plus, DNT’s are local to each DC.


 


1.) The Dave account in the forest-root domain contoso.local gets deleted.


BrandonWilson_10-1602792216389.jpeg


 


2.) The DC in contoso.local replicates that deletion to the child domain GC by telling it to delete DNT 3830.


BrandonWilson_11-1602792216393.jpeg


 


3.) The non-GC’s in the child domain don’t have the Dave account with a DNT of 3830. Instead, they have a phantom object that represents Dave with a DNT of 5585. Consequently, the Dave phantom object does not get deleted.


BrandonWilson_12-1602792216398.jpeg


 


 4.) This is where the Infrastructure Master comes in. There is one IM per domain. The IM process in this child domain runs every two days and says, “Let me review my phantom objects to make sure that the actual user accounts still exist”. Under normal conditions, it would determine that the actual Dave account got deleted and would then delete the Dave phantom object from itself and then replicate that to other DC’s in the child domain that aren’t GC’s. The problem here is though, the Infrastructure Master is running on a GC and we all know by now that GC’s doesn’t have any phantom objects. Consequently, the IM determines, “since I don’t have any phantom objects, there’s really nothing for me to do”. Therefore, the phantom object for the Dave account remains on all non-GC’s in the child domain. If you were to look at the administrators group on any of these non-GC’s in the child domain, Dave would still show as present even though the actual user account was deleted from the forest-root and replicated to all global catalog servers in the child.


Technically, the best practice should have been “Only put the Infrastructure Master on DC’s that have phantom objects” but this would have caused more confusion so Microsoft simplified it and just made it “Don’t put the Infrastructure Master on a GC”.


 


Why, Oh Why?


 


I know you’re probably thinking all of this is a convoluted way of adding users from one domain to groups in another domain, right? Well, what are all of the possible options? Let’s think about this:



  1. Allow a DC to add a user to a group even though the user account doesn’t exist in the local data table. This would break the database and referential integrity. Definitely not a good option.

  2. Don’t allow our customers to add users from one domain into groups from another domain. Once again, not a good option.

  3. Recommend that all domain controllers be global catalog servers, which negates the entire phantom object scenario. Wait a minute, we already recommend that!

  4. Create Phantom Objects on non-GC’s in other domains and then allow the Infrastructure Master to keep those phantom objects update to date, which is exactly what we’re doing today.


 


Have you thanked your infrastructure master lately? Perhaps you should.  :smile:


 


 


Chris Cartwright


 

Access Your Desktop And Applications From Anywhere

Access Your Desktop And Applications From Anywhere

by | Oct 15, 2020 | Azure, Business, Microsoft, Technology

This article is contributed. See the original author and article here.

If you are as interested in Windows Virtual Desktop and many are, Christiaan Brinkhoff does a complete walk through of how to prepare and deploy Windows Virtual Desktop. At this point, this is the best place for how to build it from the ground up: https://www.christiaanbrinkhoff.com/2020/05/01/windows-virtual-desktop-technical-2020-spring-update-arm-based-model-deployment-walkthrough/

I used this post back in early May as a guide, following it step-by-step, and found it to be very effective. Christiaan talks about the why and how and has a fantastic viewpoint (and is connected to the product).

For more information or assistance on deploying Azure virtual desktop for your business contact us.

Track Updates And Changes To Azure Services With AzureCharts

Track Updates And Changes To Azure Services With AzureCharts

by | Oct 15, 2020 | Azure, Technology

This article is contributed. See the original author and article here.

The rate of change in cloud service offerings is astounding. How do you keep up? Fortunately, there is a fantastic visual guide to tracking things here: https://azurecharts.com/


 


This is continuously updated with new service offerings and Azure updates as well a new features and functionality with the AzureCharts site itself.  If you do anything in Azure, this is an excellent resource to identify what services are available where, what’s coming next, and a multitude of other things.  Azure Charts: “Communicating Azure’s current state, structure and updates in a compact digestible way.”


 


image.png

Experiencing Alerting failure for Log Search Alerts in East US region – 10/15 – Resolved

by | Oct 15, 2020 | Technology

This article is contributed. See the original author and article here.

Final Update: Thursday, 15 October 2020 19:49 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 10/15, 19:40 UTC. Our logs show the incident started on 10/15, 18:00 UTC and that during the one hour and forty minutes that it took to resolve the issue 1.1% of customers in the East US region experienced alert failures, delayed alerts and query failures.


  • Root Cause: The failure was due to a back end component becoming unresponsive due to unexpectedly heavy load. The component self-healed.

  • Incident Timeline: 1 Hour & 40 minutes – 10/15, 18:00 UTC through 10/15, 19:40 UTC

We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Jack Cantwell
Initial Update: Thursday, 15 October 2020 19:17 UTC

We are aware of issues within Log Search Alerts and Log Analytics query and are actively investigating. Some customers may experience delayed or failed alerts as well as errors while querying Log Analytics data in the Azure portal.

  • Next Update: Before 10/15 20:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jack Cantwell

Azure Policy- Prevent The Use Of Wildcard For Source In Azure Just In Time

Azure Policy- Prevent The Use Of Wildcard For Source In Azure Just In Time

by | Oct 15, 2020 | Azure, Technology

This article is contributed. See the original author and article here.

You want to use Just In Time access for Azure VMs, but do not want the users to select all available IPs when requesting the access. Try this policy out to prevent this from happening:


 


 


 

{
    "mode": "All",
    "policyRule": {
        "if": {
            "allOf": [{
                "field": "type",
                "equals": "Microsoft.Network/networkSecurityGroups"
            }, {
                "count": {
                    "field": "Microsoft.Network/networkSecurityGroups/securityRules[*]",
                    "where": {
                        "allOf": [{
                            "field": "Microsoft.Network/networkSecurityGroups/securityRules[*].sourceAddressPrefix",
                            "equals": "*"
                        }, {
                            "field": "Microsoft.Network/networkSecurityGroups/securityRules[*].access",
                            "equals": "Allow"
                        }, {
                            "field": "Microsoft.Network/networkSecurityGroups/securityRules[*].direction",
                            "equals": "Inbound"
                        }, {
                            "field": "Microsoft.Network/networkSecurityGroups/securityRules[*].name",
                            "contains": "SecurityCenter-JITRule"
                        }, {
                            "anyOf": [{
                                "field": "Microsoft.Network/networkSecurityGroups/securityRules[*].destinationPortRange",
                                "equals": "22"
                            }, {
                                "field": "Microsoft.Network/networkSecurityGroups/securityRules[*].destinationPortRange",
                                "equals": "3389"
                            }, {
                                "field": "Microsoft.Network/networkSecurityGroups/securityRules[*].destinationPortRanges",
                                "equals": "22"
                            }, {
                                "field": "Microsoft.Network/networkSecurityGroups/securityRules[*].destinationPortRanges",
                                "equals": "3389"
                            }]
                        }]
                    }
                },
                "greater": 0
            }]
        },
        "then": {
            "effect": "deny"
        }
    },
    "parameters": {}
}

 


 


More details and comments/issues can be found here: Github: deny-wildcard-source-for-just-in-time-requests 


image.png

Vision on the Edge

Vision on the Edge

by | Oct 15, 2020 | Technology

This article is contributed. See the original author and article here.








Vision on the Edge


Introduction


Visual inspection of products, resources, and environments has been a core practice for most Enterprises, and was, until recently, a very manual process. An individual, or group of individuals, was responsible for performing a manual inspection of the asset or environment, which, depending on the circumstances, could become inefficient, inaccurate or both, due to human error and limitations.


In an effort to improve the efficacy of visual inspection, Enterprises began turning to deep learning artificial neural networks known as convolutional neural networks, or CNNs, to emulate human vision for analysis of images and video. Today this is commonly called computer vision, or simply Vision AI. Artificial Intelligence for image analytics spans a wide variety of industries, including manufacturing, retail, healthcare, and the public sector, and an equally wide area of use cases.


 


Vision as Quality Assurance – In manufacturing environments, quality inspection of parts and processes with a high degree of accuracy and velocity is one of the use cases for Vision AI. An enterprise pursuing this path automates the inspection of a product for defects to answer questions such as:



  • Is the manufacturing process producing consistent results?

  • Is the product assembled properly?

  • Can I get notification of a defect sooner to reduce waste?

  • How can I leverage drift in my computer vision model to prescribe predictive maintenance?


Vision as Safety – In any environment, safety is a fundamental concern for every Enterprise on the planet, and the reduction of risk is a driving force for adopting Vision AI. Automated monitoring of video feeds to scan for potential safety issues affords critical time to respond to incidents, and opportunities to reduce exposure to risk. Enterprises looking at Vision AI for this use case are commonly trying to answer questions such as:



  • How compliant is my workforce with using personal protective equipment?

  • How often are people entering unauthorized work zones?

  • Are products being stored in a safe manner?

  • Are there non-reported close calls in a facility, i.e. pedestrian/equipment “near misses?”


Why vision on the Edge


Over the past decade, computer vision has become a rapidly evolving area of focus for Enterprises, as cloud-native technologies, such as containerization, have enabled portability and migration of this technology toward the network edge. For instance, custom vision inference models trained in the Cloud can be easily containerized for use in an Azure IoT Edge runtime-enabled device.


The rationale behind migrating workloads from the cloud to the edge for Vision AI generally falls into two categories – performance and cost.


On the performance side of the equation, exfiltrating large quantities of data can cause an unintended performance strain on existing network infrastructure. Additionally, the latency of sending images and/or video streams to the Cloud to retrieve results may not meet the needs of the use case. For instance, a person straying into an unauthorized area may require immediate intervention, and that scenario can ill afford latency when every second counts. Positioning the inferencing model near the point of ingest allows for near-real-time scoring of the image, and alerting can be performed either locally or through the cloud, depending on network topology.


In terms of cost, sending all of the data to the Cloud for analysis could significantly impact the ROI of a Vision AI initiative. With Azure IoT Edge, a Vision AI module could be designed to only capture the relevant images that have a reasonable confidence level based on the scoring, which significantly limits the amount of data being sent.


The purpose of this document is to give some concrete guidance on some of the key decisions when designing an end-to-end vision on the edge solution. Specifically, we will address:



Camera Considerations


Camera selection


One of the most critical components to any vision workload is selecting the correct camera. The items that are being identified in a vision workload must be presented in such a way so that a computer’s artificial intelligence or machine learning models can evaluate them correctly. To further understand this concept, you need to understand the different camera types that can be used. One thing to note in this article as we move forward, there are a lot of different manufacturers of Area, Line, and Smart Cameras. Microsoft does not recommend any vendor over another – instead, we recommend that you select a vendor that fits your specific needs.


Area Scan Cameras


This is more your traditional camera image, where a 2D image is captured and then sent over to the Edge hardware to be evaluated. This camera typically has a matrix of pixel sensors.


When should you use an Area Scan Camera? As the name suggests, Area Scan Cameras look at a large area and are great at detecting change in an area. Some examples of workloads that would use an Area Scan Camera would be workplace safety or detecting or counting objects (people, animals, cars,etc.) in an environment.


Examples of manufacturers of Area Scan Cameras are Basler, Axis, Sony, Bosch, FLIR, Allied Vision


Line Scan Cameras


Unlike the Area Scan Cameras, the Line Scan Camera has a single row of linear pixel sensors. This can allow the camera to take one-pixel width in very quick successions and then stitches these one-pixel images into a video stream that is sent over to an Edge Device for processing


When should you use a Line Scan Camera? Line Scan Cameras are great for vision workloads where the items to be identified are moving past the camera or items that need to be rotated to detect defects. The Line Scan Camera would then be able to produce a continuous image stream that can then be evaluated. Some examples of workloads that would work best with a Line Scan Camera would be item defect detection on parts that are moved on a conveyer belt, workloads that require spinning to see a cylindrical object or any workload that requires rotation.


Examples of manufacturers of Area Scan Cameras are Basler, Teledyne Dalsa, Hamamatsu Corporation, DataLogic, Vieworks, and Xenics


Embedded Smart Camera


This type of camera can use either a Area Scan or Line Scan Camera for the acquisition of the images, however, the Line Scan Smart Camera is rare. The main feature of this camera is that it not only acquires the image, but it can also process the image as they are a self-contained stand-alone system. They typically have either and RS232 or Ethernet port output, and this allows the Smart Cameras to be integrated directly into a PLC or other IIoT interfaces.


Examples of manufacturers of Embedded Smart Cameras are Basler, Lesuze Electronics


Other camera features to consider



  • Sensor size: This is one of the most important factors to evaluate in any vision workload. A sensor is the hardware within a camera that is capturing the light and converting into signals which then produces an image. The sensor contains millions of semiconducting photodetectors that are called photosites. One thing that is a bit of a misconception is that a higher megapixel count is a better image. For example, let’s look at two different sensor sizes for a 12-megapixel camera. Camera A has a ½ inch sensor with 12 million photosites and camera B has a 1-inch sensor with 12 million photosites. In the same lighting conditions, the camera that has a 1-inch sensor will be cleaner and sharper. Many cameras that would be typically be used in vision workloads would have a sensor between ¼ inch to 1 inch. In some cases, much larger sensors might be required. If a camera has a choice between a larger sensor or a smaller sensor some factors consider as to why you might choose the larger sensor are:

    • need for precision measurements

    • Lower light conditions

    • Shorter exposure times, i.e. fast-moving items



  • Resolution: This is another very important factor to both Line Scan and Area Scan camera workloads. If your workload must identify fine features (Ex. writing on an IC Chip) then you need greater resolutions of the cameras used. If your workload is trying to detect a face, then higher resolution is required. And if you need to identify a vehicle from a distance, again this would require higher resolution.

  • Speed: Sensors come in two types a CCD and a CMOS. If the vision workload requires high number of images per second capture rate, then there are two factors that come into play. The first is how fast is the connection on the interface of the camera and the second is what type of sensor is it. CMOS sensors have a direct readout from the photosites and because of this they typically offer a higher frame rate.



NOTE: There are more camera features to consider when selecting the correct camera for your vision workload. These include lens selection, focal length, monochrome, color depth, stereo depth, triggers, physical size, and support. Sensor manufacturers can help you understand the specific feature that your application may require.



Camera Placement (location, angle, lighting, etc.)


Depending on the items that you are capturing in your vision workload will determine the location and angles that the camera should be placed. The camera location can also affect the sensor type, lens type, and camera body type. There are several key concepts to keep in mind when figuring out the perfect spot to place the camera in.


There are several different factors that can weigh into the overall decision for camera placement. Two of the most critical are lighting and field of view


Camera Lighting


In a computer vision workload, lighting is a critical component to camera placement. There are several different lighting conditions. While some of the lighting conditions would be useful for one vision workload, it might produce an undesirable condition in another. Types of lighting that are commonly used in computer vision workloads are:




  • Direct lighting: This is the most commonly used lighting condition. This light source is projected at the object to be captured for evaluation.




  • Line lighting: This is a single array of lights that are most used with line scan camera applications to create a single line of light where the camera is focused.




  • Diffused lighting: This type of lighting is used to illuminate an object but prevent harsh shadows and is mostly used around specular objects.




  • Back lighting: This type of light source is used behind the object, in which produces a silhouette of the object. This is most useful when taking measurements, edge detection, or object orientation.




  • Axial diffused lighting: This type of light source is often used with highly reflective objects, or to prevent shadows on the part that will be captured for evaluation.




  • Custom Grid lighting: This is a structured lighting condition that lays out a grid of light on the object, the intent is to have a known grid projection to then provide more accurate measurements of components, parts, placement of items, etc.




  • Strobe lighting: Strobe lighting is used for high speed moving parts. The strobe must be in sync with the camera to take a “freeze” of the object for evaluation, this lighting helps to prevent motion blurring effect.




  • Dark Field lighting: This type of light source uses several lights in conjunction with different angles to the part. For example, if the part is laying flat on a conveyor belt the lights would be placed at a 45-degree angle to the part. This type of lighting is most useful when looking at highly reflective clear objects…and is most commonly used with lens scratch detections.


    Angular placement of light


     

    image.png




Field of View


In a vision workload you need to know the distance to the object that you are trying to evaluate. This also will play a part in the camera selection, sensor selection, and lens configuration. Some of the components that make up the field of view are:



  • Distance to object(s): For an example is the object that we are monitoring with computer vision on a conveyor belt and the camera is 2 feet above it, or is the object across a parking lot? As the distance changes so does the camera’s sensors and lens configurations.


  • Area of coverage: is the area that the computer vision trying to monitor small or large? This has direct correlation to the camera’s resolution overall, lens, and sensor type.

  • Direction of the Sun: if the computer vision workload is outside, such as monitoring a job construction site for worker safety, will the camera be pointed in the sun at any time? Keep in mind that if the sun is casting a shadow over the object that the vision workload is monitoring, items might be obscured a bit. Also, if the camera is getting direct sunlight in the lens, the camera might be “blinded” until the angle of the sun changes.

  • Camera angle to the object(s): angle of the camera to the object that the vision workload is monitoring is also critical component to think about. If the camera is too high it might miss the details that the vision workload is trying to monitor for, and the same may be true if it is too low.


Communication Interface


In building a computer vision workload it is also important to understand how the system will interact with the output of the camera. Below are a few of the standard ways that a camera will communicate to IoT Edge:




  • Real Time Streaming Protocol(RTSP): RTSP is a protocol that transfers real-time video data from a device (in our case the camera) to an endpoint device (Edge compute) directly over a TCP/IP connection. It functions in a client server application model that is at the application level in the network.




  • Open Network Video Interface Forum (ONVIF): a global and open industry forum that is developing open standards for IP-based cameras. This standard is aimed at standardization of communication between the IP Camera and down stream systems, Interoperability, and Open sourced.




  • USB: Unlike RTSP and ONVIF USB connected cameras connect over the Universal Serial Bus directly on the Edge compute device. This is less complex; however, it is limited on distance that the camera can be placed away from the Edge compute.




  • Camera Serial Interface: CSI specification is from Mobile Industry Processor Interface(MIPI). It is an interface that describes how to communicate between a camera and a host processor.




There are several standards defined for CSI



  • CSI-1: This was the original standard that MIPI started with.

  • CSI-2: This standard was released in 2005, and uses either D-PHY or C-PHY as physical layers options. This is further divided into several layers:

    1. Physical Layer (C-PHY, D-PHY)

    2. Lane Merger layer

    3. Low Level Protocol Layer

    4. Pixel to Byte Conversion Layer

    5. Application layer




The specification was updated in 2017 to v2, which added support for RAW-24 color depth, Unified Serial Link, and Smart Region of Interest.


Hardware Acceleration


Along with the camera selection, one of the other critical decisions in Vision on the Edge projects is hardware acceleration. Options include:



  • CPU: The Central Processing Unit (CPU) is your default compute for most processes running on a computer, it is designed for general purpose compute. Some Vision Workloads where timing is not as critical this might be a good option. However, most workloads that involve critical timing, multiple camera streams, and/or high frame rates will require more specific hardware acceleration

  • GPU: Many people are familiar with the Graphics Processing Unit (GPU) as this is the de-facto processor for any high-end PC graphics card. In recent years the GPU has been leveraged in high performance computer (HPC) scenarios, and in data mining, and in computer AI/ML workloads. The GPU’s massive potential of parallel computing can be used in a vision workload to accelerate the processing of pixel data. The downside to a GPU is its higher power consumption, which is a critical factor to consider for your vision workload.

  • FPGA: Field Programmable Gate Arrays are reconfigurable hardware accelerators. These powerful accelerators allow for the growth of Deep Learning Neural networks, which are still evolving. These accelerators have millions of programmable gates, hundred of I/O pins, and exceptional compute power (in the Trillions of tera-MAC’s) There also a lot of different libraries available for FPGA’s to use that are optimized for vision workloads. Some of these libraries also include preconfigured interfaces to connect to downstream cameras and devices. One area that FPGA’s tend to fall short on is floating point operations, however, manufacturers are currently working on this issue and have made a lot of improvements in this area.

  • ASIC: Application Specific Integrated Circuit is by far the fastest accelerator on the market today. While they are the fastest, they are the hardest to change as they are manufactured to function for a specific task. These custom chips are gaining popularity due to size, power per watt performance, and IP protection (because the IP is burned into the ASIC accelerator it is much harder to backwards engineer proprietary algorithms).


Machine learning and data science


The process of designing the machine learning (ML) approach for a vision on the edge scenario one of the biggest challenges in the entire planning process. Therefore, it is important to understand how to consider and think about ML in the context of edge devices. Some of the considerations and hurdles are outlined below to help begin to think in terms of using machine learning to address business problems and pain points with guidance including:



  • Always consider first how to solve the problem without ML or with a simple ML algorithm

  • Have a plan to test several ML architectures as they will have different capacities to “learn”

  • Have a system in place to collect new data from the device to retrain an ML model

  • For a poorly performing ML models, often a simple fix is to add more representative data to the training process and ensure it has variability with all classes represented equally

  • Remember, this is often an iterative process with both the choice of data and choice of architecture being updated in the exploratory phase

  • More guidance below


It is not an easy space and, for some, a very new way of thinking. It is a data driven process. Careful planning will be critical to successful results especially on very constrained devices.


In ML it is always critical to clearly define the problem trying to be solved because the data science and machine learning approach will depend upon this and decisions will be easier the more specific it is. It is also very important to consider what type of data will be encountered in the edge scenario as this will determine the kind of ML algorithm that should be used.


Even at the start, before training any models, real world data collection and examination will help this process greatly and new ideas could even arise. Below, we will discuss data considerations in detail. Of course, the equipment itself will help determine the ML approach with regard to device attributes like limited memory, compute, and/or power consumption limits.


Fortunately, data science and machine learning are iterative processes, so if the ML model has poor performance, there are many ways to address issues through experimention. Below, we will discuss consideratinos around ML architecture choices. Often, there will be some trial and error involved as well.


Machine learning data


Both the source(s) and attributes of data will dictate how the intelligent edge system is built. For vision, it could be images, videos, or even LiDAR, as the streaming signal. Regardless of the signal, when training an ML model and using it to score new data (called inferencing) domain knowledge will be required, such as experience in designing and using ML algorithms or neural network architectures and expertise deploying them to the specialized hardware. Below are a few considerations related to ML, however, it is recommended to gain some deeper knowledge in order to open up more possibilities or find an ML expert with edge experience to help with the project.


Collecting and using a balanced dataset is critical, that is, equally representating all classes or categories. When the ML model is trained on a dataset, generally that dataset has been split into train, validate and test subsets. The purpose of these subsets is as follows.



  • The training dataset is used for the actual model training over many passes or iterations (often called epochs).

  • Througout the training process, the model is spot-checked for how well it is doing on the validation dataset.

  • After a model is done training, the final step is to pass the test dataset through it and assess how well it did as a proxy to the real-world. Note: be wary of optimizing for the test dataset (in addition to the training dataset) once one test has been run. It might be good to have a few different test datasets available.


Some good news is if using deep learning, often costly and onerous feature engineering, featurization, and preprocessing can be avoided because of how deep learning works to find signal in noise better than traditional ML. However, in deep learning, transformations may still be utilized to clean or reformat data for model input during training as well as inference. Note, the same preprocessing needs to be used in training and when the model is scoring new data.


When advanced preprocessing is used such as de-noising, adjusting brightness or contrast, or transformations like RGB to HSV, it must be noted that this can dramatically change the model performance for the better or, sometimes, for the worse. In general, it is part of the data science exploration process and sometimes it is something that must be observed once the device and other components are placed in a real-world location.


After the hardware is installed into its permanent location, the incoming data stream should be monitored for data drift.



  • Data drift: deviation due to changes in the current data compared to the original. Data drift will often result in a degradation in in model performance (like accuracy), albeit, this is not the only cause of decreased performance (e.g. hardware or camera failure).


There should be an allowance for data drift testing in the system. This new data should also be collected for another round of training (the more representative data collected for training, the better the model will perform in almost all cases!), therefore, preparing for this kind of collection is always a good idea.


In addition to using data for training and inference, new data coming from the device could be used to monitor the device, camera or other components for hardware degradation.


In summary, here are the key considerations:



  • Always use a balanced dataset with all classes represented equally

  • The more representative data used to train a model, the better

  • Have a system in place to collect new data from device to retrain

  • Have a system in place to test for data drift

  • Only run a test set through a new ML model once – if you iterate and retest on the same test set this could cause overfitting to the test set in addition to the training set


Machine learning architecture choices


An ML architecture is the layout of the mathematical operations that process input into our desired, actionable output. For instance, in deep learning this would be the number of layers and neurons in each layer of a deep neural network, plus their arrangement. It is important to note that there is no guarantee that the performance metric goal (e.g. high enough accuracy) for one ML architecture will be achieved. To mitigate this, several different architectures should be considered. Often, two or three different architectures are tried before a choice is made. Remember, this is often an iterative process with both the choice of data and choice of architecture being updated in the exploratory phase of the development process.


It helps to understand the issues that can arise when training an ML model that may only be seen after training or, even, at the point of inferencing on device. Some such issues include overfitting and underfitting as introduced below.


In the training and testing process, one should keep an eye out for overfitting and underfitting:




  • Overfitting: can give a false sense of success because the performance metric (like accuracy) might be very good when the input data looks like the training data. However, overfitting can occur when the model fits to the training data too closely and can not generalize well to new data. For instance, it may become apparent that the model only performs well indoors because the training data was from an indoor setting. This can be caused by:



    • The model learned to focus on incorrect, non-representative features specifically found in the training dataset

    • The model architecture may have too many learnable parameters (correlated to the number of layers in a neural network and units per layer) – note, the model’s memorization capacity is determined by the number of learnable parameters

    • Not enough complexity or variation in the training data

    • Trained over too many iterations

    • Other reasons for good performance in training and significantly worse performance in validation and testing




  • Underfitting: the model has generalized so well that it can not tell the difference between classes with confidence – e.g. the training loss will still be unacceptably high. This can be caused by:



    • Not enough samples in training data

    • Trained for too few iterations – too generalized

    • Other reasons related to the model not being able to recognize any objects or poor recogntion and loss values during training (the assessment values used to direct the training process through a process called optimization and weight updates)




There is a trade-off between too much capacity (a large network or one with big number of learnable parameters) and too little capacity. In transfer learning (where some network layers are set as not trainable, i.e. frozen) increasing capacity would equate to “opening up” more, earlier layers in the network versus only using the last few layers in training (with the rest remaining frozen).


There isn’t a hardfast rule for determining number of layers for deep neural networks, thus sometimes several model architectures must be evaluated within an ML task. However, in general, it is good to start with fewer layers and/or parameters (“smaller” networks) and gradually increase the complexity.


Some considerations when coming up with the best architecture choice will include the inference speed requirements which will need to include an assessment and acceptance of the speed versus accuracy tradeoff. Often, a faster inference speed is associated with lower performance (e.g. accuracy, confidence or precision could suffer).


A discussion around requirements for the ML training and inferencing will be necessary based upon the considerations above and any company specific requirements. For instance, if the company policy allows open source solutions to be utilized, it will open up a great deal of ML algorithmic possibilities as most cutting edge ML work is in the open source domain.


In summary, here are the key considerations:



  • Keep an eye out for overfitting and underfitting

  • Testing several ML architectures is often a good idea – this is an iterative process

  • There will be a trade-off between too much network capaticy and too little, but often it’s good to start with too little and build up from there

  • There will be a trade-off between speed and your performance metric (e.g. accuracy)

  • If the performance of the ML model is acceptable, the exploratory phase is complete (one can be tempted to iterate indefinitely)


Data science workflows


The data science process for edge deployments has a general pattern. After a clear data-driven problem statement is formulated, the next steps generally include the following.


 

 



  • Data Collection. Data collection or acquisition could be an online image search, from a currently deployed device, or other representative data source. Generally, the more data the better. In addition, the more variability, the better the generalization.

  • Data Labeling. If only hundreds of images need to be labeled usually (e.g. when using transfer learning) this is done in-house, whereas, if tens of thousands of images need to be labeled, a vendor could be enlisted for both data collection and labeling.

  • Train a Model with ML Framework. An ML framework such as TensorFlow or PyTorch (both with Python and C++ APIs) will need to be chosen, but usually this depends upon what code samples are available in open source or in-house, plus experience of the ML practitioner. Azure ML may be used to train a model using any ML framework and approach – it is agnostic of framework and has Python and R bindings, plus many wrappers around popular frameworks.

  • Convert the Model for Inferencing on Device. Almost always, a model will need to be converted to work with a particular runtime (model conversion usually involves advantageous optimizations like faster inference and smaller model footprints). This step differs for each ML framework and runtime, but there are open-source interoperability frameworks available such as ONNX and MMdnn.

  • Build the Solution for Device. The solution is usually built on the same type of device as will be used in the final deployment because binary files are created that are system specific.

  • Using Runtime, Deploy Solution to Device. Once a runtime has been chosen (that is usually chosen in conjunction with ML framework choice), the compiled solution may be deployed. The Azure IoT Runtime is a Docker-based system in which the ML runtimes may be deployed as containers.


The diagram below gives a picture with an example data science process wherein open source tools may be leveraged for the data science workflow. Data availability and type will drive most of the choices, even, potentially, the devices/hardware chosen.


image.png


If a workflow is already in existance for the data scientists and app developers, a few other considerations exist. First, it is advised to have a code, model and data versioning system in place. Secondly, an automation plan for code and integration testing along with other aspects of the data science process (triggers, build/release process, etc.) will help speed up time to production and cultivate collaboration within the team.


The language of choice can help dictate what API or SDK is used for inferencing and training ML models which will then dictate what type of ML model, what type(s) of device, what type of IoT Edge Module, etc. For example, PyTorch has a C++ API for inferencing (and now for training) that works well in conjunction with the OpenCV C++ API. If the app developer working on the deployment strategy is building a C++ application, or has this experience, one might consider PyTorch or others (TensorFlow, CNTK, etc.) that have C++ inferencing APIs.


In summary, here are the key considerations:



  • Converting models also involves optimizations such as faster inference and smaller model footprints, critical for very resource-constrained devices

  • The solution will usually need to be built on a build-dedicated device (the same type of device to which the solution will be deployed)

  • The language and framework of choice will depend upon both the ML practitioners experience as well as what is available in open source

  • The runtime of choice will depend upon the device and hardware acceleration for ML available

  • It is important to have a code, model and data versioning system


Image storage and management


Storage and management of the images involved in a computer vision application is a critical function. Some of the key considerations for managing those images are:



  • Ability to store all raw images during training with ease of retrieval for labeling

  • Faster storage medium to avoid pipeline bottleneck and loss

  • Storage on the edge as well as in the cloud, as labeling activity can be performed in both

  • Categorization of images for easy retrieval

  • Naming and tagging images to link it with inferred metadata


The combination of Azure Blob Storage, Azure IoT Hub, and Azure IoT Edge allow several potential options for the storage of image data:



  • Use of the Azure IoT Edge Blob Storage module, which will automatically sync images to Azure Blob based on policy

  • Store images to local host file system and upload to Azure blob service using a custom module

  • Use of local database to store images, which then can be synced to cloud database


We believe that the IoT Edge Blob Storage module is the most powerful and straightforward solution and is our preferred approach. A typical workflow for this might be:



  1. Raw messages post ingestion will be stored locally on the Edge Blob Module, with time stamp and sequence number to uniquely identify the image files

  2. Policy can be set on the Edge Blob Module for automatic upload to Azure Blob with ordering

  3. To conserve space on the Edge device, auto delete after certain time can be configured along with retain while uploading option to ensure all images get synced to the cloud

  4. Local categorization or domain and labeling can be implemented using module that can read these images into UX. The label data will be associated to the image URI along with the coordinates and category.

  5. As Label data needs to be saved, a local database is preferred to store this metadata as it will allow easy lookup for the UX and can be synced to cloud via telemetry messages.

  6. During scoring run, the model will detect matching patterns and generate events of interest. This metadata will be sent to cloud via telemetry referring the image URI and optionally stored in local database for edge UX. The images will continue to be stored to Edge Blob and synced with Azure Blob


Alerts persistence


In the context of vision on edge, alerts is a response to an event that is triggered by the AI model (in other words, the inferencing results). The type of event is determined by the training imparted to the model. These events are separate from operational events raised by the processing pipeline and any related to the health of the runtime.


Some of the common alerts types are:



  • Image classification

  • Movement detection

  • Direction of movement

  • Object detection

  • Count of objects

  • Total Count of objects over period of time

  • Average Count of objects over period of time


Alerts by their definition are required to be monitored as they drive certain actions. They are critical to operations, being time sensitive in terms of processing and required to be logged for audit and further analysis.


The persistence of alerts needs to happen locally on the edge where it is raised and then passed on to the cloud for further processing and storage. This is to ensure quick response locally and avoid losing critical alerts due to any transient failures.


Some options to achieve this persistence and cloud syncing are:



  • Utilize built-in store and forward capability of IoT Edge runtime, which automatically gets synced with Azure IoT Hub in case of losing connectivity

  • Persist alerts on host file system as log files, which can be synced periodically to a blob storage in cloud

  • Utilized Azure Blob Edge module, which will sync this data to Azure Blob in cloud based on policies that can be configured

  • Use local database on IoT Edge, such as SQL Edge for storing data, sync with Azure SQL DB using SQL Data Sync. Other lightweight database option is SQLite


The preferred option is to use the built-in store and forward capability of IoT Edge runtime. This is more suitable for the alerts due to its time sensitivity,typically small messages sizes, and ease of use.


User Interface


The user interface requirements of an IoT solution will vary depending on the overall solution objectives. In general, there are four user interfaces that are commonly found on IoT solutions: Administrator, Operator, Consumer and Analytics. In this guidance, we are going to focus on simple operator’s user interface and visualization dashboard. We will provide a reference implementation of the latter two



  • Administrator: Allows full access to device provisioning, device and solution configuration, user management etc. These features could be provided as part of one solution or as separate solutions.

  • Consumer: Only applicable to consumer solution. They provide similar access to the operators’ solution but limited to devices owned by the user

  • Operator: Provides centralize access to the operational components of the solutions which typically includes device management, alerts monitoring and configuration.

  • Analytics: Interactive dashboard which provide visualization of telemetry and other data/analysis.


Technology Options


Power BI is a compelling option for our Analytics/Virtualization needs. It provides power features to create customizable interactive dashboards. It also allows connectivity to many popular database systems and services. It is available as a managed service and as a self-hosted package. The former is the most popular and recommend options. With Power BI embedded you could add customer-facing reports, dashboards, and analytics in your own applications by using and branding Power BI as your own. Reduce developer resources by automating the monitoring, management, and deployment of analytics, while getting full control of Power BI features and intelligent analytics.


Another suitable technology for IoT visualizations is Azure Maps which allows you to create location-aware web and mobile applications using simple and secure geospatial services, APIs, and SDKs in Azure. Deliver seamless experiences based on geospatial data with built-in location intelligence from world-class mobility technology partners.


Azure App Service is a managed platform with powerful capabilities for building web and mobile apps for many platforms and mobile devices. It allows developers to quickly build, deploy, and scale web apps created with popular frameworks .NET, .NET Core, Node.js, Java, PHP, Ruby, or Python, in containers or running on any operating system. You can also meet rigorous, enterprise-grade performance, security, and compliance requirements by using the fully managed platform for your operational and monitoring tasks.


For real time data reporting, Azure SignalR Service, makes adding real-time communications to your web application is as simple as provisioning a service—no need to be a real-time communications guru! It easily integrates with services such as Azure Functions, Azure Active Directory, Azure Storage, Azure App Service, Azure Analytics, Power BI, IoT, Cognitive Services, Machine Learning, and more. To secure your user interface solutions, the Azure Active Directory (Azure AD) enterprise identity service provides single sign-on and multi-factor authentication to help protect your users from 99.9 percent of cybersecurity attacks.


Scenarios


Use case 1


Overview


Contoso Boards produces high quality circuit boards used in computers. Their number one product is a motherboard. Lately they have been seeing an increase in issues with chip placement on the board. Through their investigation they have noticed that the circuit boards are getting placed incorrectly on the assembly line. They need a way to identify if the circuit board is placed on the assembly line correctly. The data scientist at Contoso Boards are most familiar with TensorFlow and would like to continue using it as their primary ML model structure. Contoso Boards has several assembly lines that produce these mother boards. Contoso Boards would also like to centralized management of the entire solution.


Questions


What are we analyzing?



  • Motherboard


Where are we going to be viewing the motherboard from?



  • Assembly Line Conveyor belt


What camera do we need?



  • Area or Line scan

  • Color or Monochrome

  • CCD or CMOS Sensor

  • Global or rolling shutter

  • Frame Rate

  • Resolution


What type of lighting is needed?



  • Backlighting

  • Shade

  • Darkfield


How should the camera be mounted?



  • Top down

  • Side view

  • Angular


What hardware should be used?



  • CPU

  • FPGA

  • GPU

  • ASIC


Solution


Based on the overall solution that the Contoso Boards is looking for with this vision use case we should be looking for edge detection of the part. Based on this we need to position a camera directly above the at 90 degrees and about 16 inches above the part. Since the conveyer system moves relatively slowly, we can use an Area Scan camera with a Global shutter. For this use case our camera should capture about 30 frames per second. As for the resolution using the formula of Res=(Object Size) Divided by (details to be captured). Based on the formula Res=16”/8” give 2MP in x and 4 in y so we need a camera capable of 4MP. As for the sensor type, we are not fast moving, and really looking for an edge detection, so a CCD sensor could be used, however a CMOS sensor will be used. One of the more critical aspects for any vision workload is lighting. In this application Contoso Boards should choose to use a white diffused filter back light. This will make the part look almost black and have a high amount of contrast for edge detection. When it comes to color options for this application it is better to be in black and white, as this is what will yield the sharpest edge for the detection AI model. Looking at what kind of hard, the data scientist are most familiar with TensorFlow and learning ONNX or others would slow down the time for development of the model. Also because there are several assembly lines that will use this solution, and Contoso Boards would like a centrally managed edge solution so Azure Stack Edge (with GPU option) would work well here. Based on the workload, the fact that Contoso Boards already know TensorFlow, and this will be used on multiple assembly lines, GPU based hardware would be the choice for hardware acceleration.


Sample of what the camera would see


image.png


Use Case 2


Overview


Contoso Shipping recently has had several pedestrian accidents at their loading docks. Most of the accidents are happening when a truck leaves the loading dock, and the driver does not see a dock worker walking in front of the truck. Contoso Shipping would like a solution that would watch for people, predict the direction of travel, and warn the drivers of potential dangers of hitting the workers. The distance from the cameras to Contoso Shipping’s server room is to far for GigE connectivity, however, they do have a large WIFI mesh that could be used for connectivity. Most of the data scientist that Contoso Shipping employ are familiar with Open-VINO and they would like to be able to reuse the models on additional hardware in the future. The solution will also need to ensure that devices are operating as power efficiently as possible. Finally, Contoso Shipping needs a way to manage the solution remotely for updates.


Questions


What are we analyzing?



  • People and patterns of movement


Where are we going to be viewing the people from?



  • The loading docks are 165 feet long

  • Cameras will be placed 17 feet high to keep with city ordnances.

  • Cameras will need to be positioned 100 feet away from the front of the trucks.

  • Camera focus will need to be 10 feet behind the front of the truck, and 10 additional feet in front of the truck, giving us a 20 foot depth on focus.


What camera do we need?



  • Area or Line scan

  • Color or Monochrome

  • CCD or CMOS Sensor

  • Global or rolling shutter

  • Frame Rate

  • Resolution


What type of lighting is needed?



  • Backlighting

  • Shade

  • Darkfield


What hardware should be used?



  • CPU

  • FPGA

  • GPU

  • ASIC


How should the camera be mounted?



  • Top down

  • Side view

  • Angular


Solution


Based on the distance of the loading dock size Contoso Shipping will require several cameras to cover the entire dock. Based on zoning laws that Contoso Shipping must adhere to require that the surveillance cameras cannot be mounted higher that 20 feet. In this use case the average size of a worker is 5 foot 8 inches. The solution must use the least number of cameras as possible.


Formula:


image.png


For an example if we look at the following images:


Taken with 480 horizontal pixels at 20 foot


image.png


Taken with 5184 horizontal pixels at 20 foot


image.png


The red square is shown to illustrate one pixel color.


Note: This is the issue with using the wrong resolution camera for a given use case. Lens can impact the FOV, however, if the wrong sensor is used for that given use case the results could be less than expected.


With the above in mind, when choosing a camera for the overall solution required for Contoso Shipping, we need to think about how many cameras and at what resolution is needed to get the correct amount of details to detect a person. Since we are only trying to identify if a person is in the frame or not, our PPF does not need to be around 80 (which is what is about needed for facial identification) and we can use somewhere around 15-20. That would place the FOV around 16 feet. A 16-foot FOV would give us about 17.5 pixels per foot…which fits within our required PPF of 15-20. This would mean that we need a 10MP camera that has a horizontal resolution of ~5184 pixels, and a lens that would allow for a FOV of 16 feet. When looking at the solution the cameras would need to be placed outside, and the choice of sensor type should not allow for “bloom”. Bloom is when light hits the sensor and overloads the sensor with light…this causes a view of almost over exposure or a “white out” kind of condition. CMOS is the choice here. Contoso operates 24×7 and as such needs to ensure that nighttime personal are also protected. When looking at color vs Monochrome, Monochrome handles low light conditions much better, and we are not looking to identify a person based on color monochrome sensors are a little cheaper as well. How many cameras will it take? Since we have figured out that our cameras can look at a 16 foot path, it is just simple math. 165 foot dock divided by 16 foot FOV gives us 10.3125 cameras. So the solution would need 11 Monochrome 5184 horizontal pixel (or 10MP) CMOS cameras with IPX67 housings or weather boxes. The cameras would be mounted on 11 poles 100 feet from the trucks at 17f high. Based on the fact that the data scientist are more familiar with Open-VINO data models should be built in ONNX. When looking for what hardware should be used, they need a device that can be connected over WIFI, and use as little power as possible. Based on this they should look to an FPGA processor. Potentially an ASIC processor could also be utilized, but due to the nature of how an ASIC processor works, it would not meet the requirement of being able to use the models on different hardware in the future.


image.png


 


For more and updates to this project, see our GitHub repo here: https://github.com/AzureIoTGBB/iot-edge-vision








From Manila To Copenhagen: Marilag’s Mission To Upskill Tech Talent

From Manila To Copenhagen: Marilag’s Mission To Upskill Tech Talent

by | Oct 15, 2020 | Technology

This article is contributed. See the original author and article here.

Marilag Dimatulac is one community leader undeterred by distance.

 

Based in Denmark and hailing from The Philippines, The Microsoft Azure MVP does not let the more than 10,000 kilometres get in the way of her commitment to raising up tech talent in both her new home and home country.

 

Marilag is the community organizer of Azure User Group Manila and Azure User Group Copenhagen, deftly upskilling the careers of developers from opposite sides of the world at the same time. The Cloud Architect and full-stack developer specializing in Azure & .Net says community is important for developers and developing. In fact, Marilag says community has changed her approach to developing altogether.

 

“In a way, community challenges me to improve my skills as I get to meet my talented peers,” she says. “And beyond writing code and fixing bugs, it also brings me a greater sense of purpose. The community has helped me feel happier about my job. And who doesn’t want to be a happy developer?”

 

This sense of community is in stark contrast to Marilag’s initial experience in Denmark. Marilag says she did not know many people upon arrival from The Philippines, meaning she often worked alone and struggled with technical issues by herself. “I felt disconnected and consumed to a point that I considered changing career,” Marilag admits. “Joining Azure User Group Denmark changed that for me.”

 

After attending a couple of meetups, Marilag realized that she has so much to learn and so much to share with her newfound friends. “I’m thankful that AZUGDK saved me from leaving the industry that I do love. My hope is to help others who are experiencing the same challenges I had.”

 

Now, Marilag leads not only the Copenhagen User Group but also the Manila User Group. The cloud expert hopes the expansion of local developer communities can make technology feel more accessible to both sides of the globe she calls home.

 

“Acquiring technical knowledge from people who speak your language and experience the same playing field as you do is very beneficial,” she says.

 

“With AZUG Manila, our online events are able to reach developers from remote areas of the country where the technology may not be as accessible. We also have young tech students, living outside the capital, asking for advice for their career path. And for me, this is what community is about: Giving everyone a chance to see what’s possible and encouraging them to achieve it.”

 

Community is vital, Marilag says, and it is especially important for collaborative work like software development. 

 

“I would love to help open up spaces where developers can come and feel a sense of connection, so we can stay motivated to build even during difficult times. And I’m also hoping to bring that feeling all the way across the planet, back to my hometown, where I can inspire young developers, like I once was, to be part of the global effort in using technology to improve lives.”

 

For more on Marilag, visit her Twitter @marilag

 

Marilag_speaker.jpg

 

 

Staying up to date with Azure

Staying up to date with Azure

by | Oct 15, 2020 | Azure, Technology

This article is contributed. See the original author and article here.

StayingUpToDateWithAzure.png

 

Staying “current” with the always evolving Azure platform is not an easy task. Here are some tricks and websites to make this a little bit easier:

 

Azure Learning Paths & Certifications

AzureLearningPath.png

https://docs.microsoft.com/en-us/learn/azure/ 

The learning paths are super helpful to discover and master Azure topics – they are free but still very high quality. Once you have done some learning paths, I highly recommend getting Azure certifications – looks great in the CV and helps you speak the ‘Azure language’. 

 

Azure Friday

Is it Friday again? The time of the week to stay current with short information presented from Scott Hanselman in ‘Azure Friday’! In these easy to digest 10-15 min videos Scott and his guests will present the best and latest in azure technologies and beyond.

Go subscribe to never miss an episode!

AzureFriday.png

https://azure.microsoft.com/en-us/resources/videos/azure-friday/ 

 

Podcasts

Are you learning with your ears? Travelling a lot? Then maybe subscribe to an Azure podcast to stay current – the mentioned Azure Friday is available, too.

https://channel9.msdn.com/Shows/Azure-Friday

Or ‘The Azure Podcast’ going back till 2013 with lots of Azure engineers sharing their expertise:

http://azpodcast.azurewebsites.net/ 

 

Azure Community: Events & Social Media

 

A while back I would have recommended to attend local community events, user groups or Azure Saturday’s – most of them are fully virtual and its always great to connect with like-minded people: https://azure.microsoft.com/en-us/community/events/ 

 

Additionally, follow Azure engineers, PMs, advocates on Azure – the folks are very easy to reach there: https://twitter.com/azure 

 

Azure Updates

AzureUpdate.png

https://azure.microsoft.com/en-us/updates/ 

 

Lastly, Azure updates – the central place for all new and updated Azure things. A must if you want to have the latest news, features and sometimes even retirements.

 

How about you?
You have other tricks? Things that work for you?  
Share them in the comments!

 

Hope it helps,
Max

 

Learning and Networking Days – two FREE, one-day events from the EU Collaboration Summit event team

Learning and Networking Days – two FREE, one-day events from the EU Collaboration Summit event team

by | Oct 15, 2020 | Technology

This article is contributed. See the original author and article here.

The event team behind the recent Galactic Collaboration Summit is at it again, this time with two FREE, one-day “virtual world” events:


 



  • Collab & Cloud Training Day | 11 deep-dive tutorials from Microsoft MVPs and RDs

  • European Networking Day | networking, concert, and prizes – delivered completely via VR


Both are open to all and unique in their offerings.


 


Register now, compliments of Microsoft’s The Intrazone podcast.
(use this link and receive 2 CollabCoins after registering: https://csmmt.eu/ndintrazone)


 


Learn more about the day-long events and join in both…


 


Collab & Cloud Learning Day (October 26th, 2020) 



 


The European Collaboration Summit team is, together with well-known Microsoft MVPs and Microsoft Regional Directors (RDs), delivering an online Collab & Cloud Learning Day with 11 full-day tutorials. This is pure learning – no sessions, no keynotes. The deep-dive tutorials are each six hours covering topics like the Modern Workplace, Microsoft 365, Azure, Microsoft Teams, SharePoint, and the Power Platform. Among tutorial lecturers are 8 Microsoft MVPs and 4 Microsoft Regional Directors.


 


Join in the Collab & Cloud Learning Day filled with 11 world-class, full-day tutorials (workshops)Join in the Collab & Cloud Learning Day filled with 11 world-class, full-day tutorials (workshops)


European Networking Day (October 27th, 2020)



 


In this time of pandemic, we all miss networking: new contacts, business development and fun. This is why the European Collaboration Summit team decided to have a world premiere: a dedicated European Networking Day – complete with an expo, prizes, and an exclusive music concert from the C.U.R.E Girlband – all delivered in VR, on Microsoft’s AltspaceVR platform (attendees will need either an Windows PC, or VR glasses such as Oculus or HTC Vive to join).


 


Once you’re in, the Expo with 35 exhibitors ready chat, answer questions and share what they know. As you peruse the virtual expo, you can collect virtual CollabCoins which can bring you valuable prizes (see prizes listed below). And last, there will be a discussion panel and an official virtual snowball fight – pack and throw (“Hey, no ice balls!”).


 


Join in the European Networking Day, one filled with networking, a concert, and prizes - delivered completely via VR.Join in the European Networking Day, one filled with networking, a concert, and prizes – delivered completely via VR.


Prizes for CollabCoins


 



  • 1 – HTC Vive Pro HMD (600 Eur / US$ value)

  • 2-5 – 4 x Free European Collaboration Summit or European Cloud Summit tickets, valued 300€ each

  • 6-10 – 5 x Commodore C-64 Mini Gaming Console, valued 50€/$

  • Top 50 people with the most CollabCoins: 20% price cut for the European Collaboration Summit AND the next European Cloud Summit


 


Note: In addition, 35 sponsors will be giving away prizes in the total worth of 5000€.


 


And don’t forget, you can earn two CollabCoins – compliments of The Intrazone – when you register with this link: https://csmmt.eu/ndintrazone.
Reg today and get virtually prepared!


 


Additional screens to give you a preview of the European Networking Day VR experience.Additional screens to give you a preview of the European Networking Day VR experience.


Shout out to community members Adis Jugo, Spencer Harbar and the whole #CollabSummit crew who are working hard to make them both engaging, worthwhile events – supporting and promoting the knowledge and expertise that reaffirms this: Microsoft 365 has the best tech community in the world – one that spans the virtual and the virtual reality worlds.


 


– Mark “3D” Kashman, virtual senior product manager – Microsoft


 


Mark Kashman and Echo Kitty in the third dimension.Mark Kashman and Echo Kitty in the third dimension.

Azure Defender for IoT is now in public preview

Azure Defender for IoT is now in public preview

by | Oct 15, 2020 | Azure, Technology

This article is contributed. See the original author and article here.

AdobeStock_96161096.jpeg


Summary: Agentless security for unmanaged IoT/OT devices


As industrial and critical infrastructure organizations implement digital transformation, the number of networked IoT and Operational Technology (OT) devices has greatly proliferated. Many of these devices lack visibility by IT teams and are often unpatched and misconfigured, making them soft targets for adversaries looking to pivot deeper into corporate networks.


 


Business risks include financial losses due to production downtime, corporate liability from safety and environmental incidents, and theft of sensitive intellectual property such as proprietary formulas and manufacturing processes.


 


Incorporating agentless, IoT/OT-aware behavioral analytics from Microsoft’s recent acquisition of CyberX, Azure Defender for IoT addresses these risks by discovering unmanaged IoT/OT assets, identifying IoT/OT vulnerabilities, and continuously monitoring for threats.


 


Azure Defender for IoT is now available in public preview for on-premises deployments, with the option of connecting securely to Azure Sentinel to eliminate IT/OT silos and provide a unified view of threats across both IT and OT environments. It also integrates out-of-the box with third-party tools like Splunk, IBM QRadar, and ServiceNow.


 


Introduction


Announced at Ignite 2020, Azure Defender for IoT delivers agentless security for continuously monitoring OT networks in industrial and critical infrastructure organizations.


 


You can deploy these capabilities fully on-premises without sending any data to Azure. using our new native connector to integrate IoT/OT alerts into Azure Sentinel, benefiting from the scalability and cost benefits of the industry’s first cloud-native SIEM/SOAR platform.


 


Microsoft offers a number of end-to-end IoT security solutions for managed (or “greenfield”) IoT deployments, including Azure IoT Hub, Azure Sphere and micro-agents for embedded operating systems. However,  most of today’s IoT/OT devices are “unmanaged” because they do not get provisioned, are not monitored, and lack built-in security such as agents or automated updates.


 


As a result, most IT security organizations have limited or no visibility into their OT networks. What’s more, these devices are often unpatched and misconfigured, making them soft targets for adversaries looking to pivot deeper into corporate networks.


 


Network security monitoring tools developed for IT networks are unable to address these environments because they’re blind to specialized industrial protocols (Modbus, DNP3, BACnet, etc.). They also lack an understanding of the specialized device types, applications, and machine-to-machine (M2M) behaviors in IoT/OT environments.


 


Key capabilities


Azure Defender for IoT enables IT and OT teams to auto-discover their unmanaged IoT/OT assets, identify critical vulnerabilities, and detect anomalous or unauthorized behavior — without impacting IoT/OT stability or performance.


 


Azure Defender for IoT delivers insights within minutes of being connected to the network, leveraging patented IoT/OT-aware behavioral analytics and machine learning to eliminate the need to configure any rules, signatures, or other static IOCs. To capture the traffic, it uses an on-premises network sensor deployed as a virtual or physical appliance connected to a SPAN port or tap. The sensor implements non-invasive passive monitoring with Network Traffic Analysis (NTA) and Layer 7 Deep Packet Inspection (DPI) to extract detailed IoT/OT information in real-time.


 


You also benefit from out-of-the box integration with third-party IT security tools like Splunk, IBM QRadar, and ServiceNow. Plus, it’s designed to fit right into existing OT environments, even across diverse automation equipment from all major OT suppliers (Rockwell Automation, Schneider Electric, GE, Emerson, Siemens, Honeywell, ABB, Yokogawa, etc.).


 


Integration with existing SOC workflows is key to removing IT/OT silos while delivering unified monitoring and governance across both IT and OT. To help automate this complex security challenge, we’ve also beefed up Azure Sentinel with IoT/OT-specific SOAR playbooks and


 


Combined with previous support in Azure Security Center for IoT for protecting managed IoT/OT devices connected via Azure IoT Hub, these new capabilities enable organizations to accelerate their digital transformation initiatives with a combined solution for both unmanaged and managed devices.


 


Rapid Deployment.png


 


Rapid non-invasive deployment leveraging patented IoT/OT-aware behavioral analytics, available either for on-premises or Azure-connected environments.


 


Real-time threat alerts provided by Azure Defender for IoT (examples)



  • Unauthorized device connected to the network

  • Unauthorized connection to the internet

  • Unauthorized remote access

  • Network scanning operation detected

  • Unauthorized PLC programming

  • Changes to firmware versions

  • “PLC Stop” and other potentially malicious commands

  • Device is suspected of being disconnected

  • Ethernet/IP CIP service request failure

  • BACnet operation failed

  • Illegal DNP3 operation

  • Master-slave authentication error

  • Known malware detected (e.g., WannaCry, EternalBlue)

  • Unauthorized SMB login


 


Screenshot examples.png


 


 

Azure Defender for IoT provides holistic IoT/OT security including asset discovery, vulnerability management, and continuous threat monitoring, combined with deep Azure Sentinel integration.


 


Try it now at no charge


Try Azure Defender for IoT during public preview. This version includes the agentless security provided via the integration of CyberX, plus the ability to connect to Azure Sentinel. And please give us your feedback in the IoT Security Tech Community.


 


Learn more with these educational resources