March 16 | FREE Microsoft Security Public Webinar | Diversity in Cybersecurity

by Contributed | Feb 25, 2021 | Technology

This article is contributed. See the original author and article here.

Changing behaviors and systems will never be easy, even with a blueprint. Diversity & inclusion is a process of iteration, and trial and error. It requires to take a hard look at ourselves, our perceptions, and our actions. How do we increase diversity in tech?   On March 16,  Sue Loh, a software engineer at Microsoft and author of the young adult hacker novel Raven, inspires girls and other under-represented groups to enter tech. Sue will be interviewed by Karen Dahmen, Principal Group PM in Azure Security Engineering. This webinar will be the first in a series on diversity and inclusion in cybersecurity.   Details at aka.ms/SecurityWebinars.

Fully leverage the possibilities of the dev tenant

by Contributed | Feb 25, 2021 | Technology

This article is contributed. See the original author and article here.

Now let’s imagine this. You have the developer tenant from your Microsoft 365 Developer Subscription, and you enrolled the Power Apps Community Plan with it, so you have the full range of possibilities on your dev tenant, right? 

Not quite. 

Yes, you have access to Dataverse now and yes, you can even use premium connectors and that’s cool. But you can go one step further.  

The Power Apps Community Plan gives you a Development Environment and you can add even more Trial Environments on your dev tenant. But you can’t set up new Production Environments.  

What are Production Environments and why do you want one? 

Production Environments are used for your daily business with production data. If your organization is spread over multiple countries or even continents, it makes sense to give every country its own production environment. Production Environments are backed up frequently to protect your data.  

But on your dev tenant you don’t need one, right? Because you’re just testing out and learning things, correct? Well yes, but what if you want to learn how to set up a staging environment for your organization, where you have the recommended environments? 

1. Development 

1. Testing 

1. Production 

And imagine you want to learn how to set up the correct security roles in Azure to test things out. You can’t do that on your regular dev tenant, because you are not allowed to set up more environments (especially Production Environments).  

Many users are simply not allowed to get full control in the organizations they’re working for, because of security and compliance reasons.  

So, what to do? 

Wouldn’t it be cool, if you could get a Power Apps Standalone license for your dev tenant? Sure, that costs money but you would have it all: 

• A Microsoft 365 E5 license 
• Multiple environments to plan out real-life scenarios for your organization 
• Dummy users, mailboxes and SharePoint sites to test your flows and apps 
• Premium connectors  
• Access to Dataverse 

And guess what? You can actually purchase a Power Apps standalone license for your dev tenant! You are capable to build a complete safe tenant to test, learn and grow for your organization with just the cost of the Power Apps license.  

Here is how you do it 

Log into your dev tenant and go to the Admin center. Click on the waffle menu in the upper left corner and click on Admin. 

Next, click on Purchase services. 

In the category Business apps you find all the Power Platform plans. In this example I highlighted the Power Apps per user plan, but you can also go for the Power Automate per user plan (it’s way cheaper and you have very similar possibilities. Remember, you just want to create different environments and you can do that with both plans. Then click on Details to see, what’s included. 

In the next screen you can check all the details. Click on Included apps to see every detail. If you’re happy with it click on Purchase. 

Now Now

Now you’re almost done. You already are at the Checkout. Fill in your postal address and your billing information. After that click on Place order. 

After a few minutes you will get your confirmation mail.  

If you click on Get started you will be transferred to the Your products section of the Admin center. Click on Assign licenses. 

For whatever reasons, you have to click on Assign licenses once again.  

After that you can type in a name and click on the chevron icon next to Turn apps and services on or off. Check all the apps and services you want to assign, then finish the process by clicking on Assign again, on the bottom of the page. 

BOOM, there you go.  

As described in my previous post How to enhance your “dev tenant” to unleash the full potential of the Power Platform, you can now add multiple environments and yes, even Production Environments. Check out the Microsoft documentation about environments to get more details and why and how to use them. Enjoy. 

[Guest Blog] XR and How To Bridge The Gap

by Contributed | Feb 25, 2021 | Technology

This article is contributed. See the original author and article here.

This article was written by Nick Hughes, an avid contributor in the extended reality space as part of our Humans of Mixed Reality Guest Blogger Series. Nick Hughes shares about his approach and core best practices to the Mixed Reality stack and how YOU can bring value in XR projects.

The whole extended reality space is vast, fluid, and sometimes foolish to try to define. There are so many ways that we can leverage this innovative modality that it also makes it somewhat difficult to pitch. The idea of bridging the gap between extended reality and the tangible world is still relatively fresh to most people. In this article, I’m going to give my experience taking a pragmatic approach to the Microsoft mixed reality stack and how you can bring value to your XR projects.

The first ever HoloLens I used is still in my office!

You’re probably thinking to yourself, “This guy is probably a Microsoft partner or some industry expert in mixed reality.” I assure you that two years ago I was running network cables down soot-covered walls in a foundry. 

If there is someone who has asked all the silly questions, someone who has had to search the acronym on the Internet, it is me. However, that is why what I am going to share might be helpful to you too. I am a jack-of-all-trades-master-of-none type of person who just happens to be exceptional at translating all things technology into a humanly digestible form. This is going to land right in your backyard.

Two years ago, I was first exposed to XR. I put on a friend’s PlayStation VR headset and found myself captivated with the experience. The next couple of months had me buried in articles, YouTube videos, blogs, you name it. When I found the HoloLens, I couldn’t stop imagining all the business applications and use case scenarios. I knew this was going to be transformative, but I didn’t know it was going to radically change my entire career direction as well.

Today, I help lead an extended reality team that covers the entire globe. We have deployed HoloLens and Remote Assist in nearly twenty countries and on almost every continent. This technology has dramatically transformed the way we can conduct business and I’m thrilled to have the opportunity to share with everyone how we did it.

1. SHARE THIS AS SOON AS YOU CAN
   I cannot stress to you the importance of excitement. Gordon B. Hinckley said, “Every good citizen adds to the strength of a nation” and that should be your mindset when you first get your hands on a HoloLens. Your first goal should be to create as many “good citizens” of XR as possible. These people will help you to develop the organic business excitement that will be instrumental in the adoption, peer-aided adoption, and funding – yes funding! – of your efforts.
   A word of caution though, if you post all these showy videos in public places, once that excitement gets to a certain point, you’ll have to deliver that capability. Try to be intentional and align your program sponsors before it gets to that level. They may want you to temper that excitement and keep it regulated.
   
   


2. BECOME A PART OF THE MIXED REALITY COMMUNITY
   If you’ve found us here on the Mixed Reality Tech Community, you’re off to a good start! I and most ordinary humans can only partially consume all the information that is whirring around about XR. Developers are releasing original ideas and capabilities each day. A couple of suggestions that I recommend are the Microsoft Mixed Reality Community, Twitter, and the Global XR Community. These have been very effective in building my XR wisdom and helping flatten my learning curve.
   
   Don’t be intimidated – it is going to seem foreign to you at first and that’s OK. As my friend, Steve Kaminski over at Microsoft would say, “just hang on by your fingernails to what you don’t know at first. If you stay engaged, it will all begin to make sense soon”. If some of the content doesn’t hit home, go jump into AltSpaceVR for a support group or post something in the mixed reality forums. The XR community is wonderfully accommodating and we’re all here to learn together.
   
   


3. REMOTE ASSIST, TRAINING, INTEGRATION – SAY IT WITH ME
   The most straightforward value proposition is the cost savings from eliminating travel altogether. You will crush geographical barriers by traversing over border closings and eliminating the impact of COVID travel bans with ease. This will be huge for anyone doing business globally. All three of those elements can compound into a trifecta business case that is impossible to ignore. 
   Case in point: We are getting complete returns on our Microsoft stack investments in less than six months (~ 3 months) of having a device in operation. It practically pays for itself over time! Now THAT is powerful.
   
   Training is another significant piece of business that is challenging with the current health concerns. Remote Assist and Guides both facilitate the pseudo-face-to-face training as well as virtually guided training. This eliminates the need for another person to be present and helps mitigate risk as well as reduces labor cost. The icing on the cake is that I can normally train a slow adopter of technology how to use Guides in less than an hour because it is so simple. Two weeks into our rollout, a senior manager had come into an MTC meeting and asked “What is the next big thing? What does the future look like?” I can confidently answer that with a single word, integration. It isn’t a new technology that is next, but how we use extended reality. Connect your MES systems, connect your ERP and show your “good citizens” how to advance customized applications. That is like opening Pandora’s box.
   
   


4. CAPTURE THE STORY AND MAKE IT REAL
   Microsoft, specifically, has done a great job at making sure they have enough videos and knowledge available so that some of the stories can be told without too much trouble. However, to get that essential excitement, you’re going to want to show you and your people using XR at your own locations and in your own ways. So, let me accelerate that effort for you.
   As soon as you get your HoloLens 2, grab a laptop on the same network, go to the device portal, and then to the capture section. Now, open your HoloLens and record from the HoloLens. You can capture a brilliant first-person perspective of what you’re viewing. This will be meaningful when you’re making your marketing materials, especially for your executive stakeholders.
   
   You will also want to go buy a Microsoft Display Adapter. I know, I’m sounding like a bit of a salesman for Microsoft (trust me, I wish I got commissions too!) but, this adapter is small, inexpensive, and works without any configuration. Open the box, stick it in the HDMI and USB, and magically stream the first-person video directly to the television. This comes in very convenient when you’re showing someone the HoloLens for the first time!
   
   


5. KEEP ORBITING
   If you take any advice from me it is this, “stay present and keep orbiting”. This technology will sell itself but you must keep in touch and you must be ready to seize an opportunity when it presents itself. You’ll be working quietly for a week and then someone will tell you about this new thing they want to explore! Yes! That is exactly what you want to happen.
   
   Carpe diem, friends!
   
   Do not let those kinds of opportunities go to waste because those “good citizens” are realizing the additional capability to the business. Be a facilitator and help them develop their ideas. That will resonate back to the business in a grand way. Especially when the users can build and then share their own story. You’ll see dividends from the peer-aided adoption that will come from those interactions.
   
   Remote Assist will consistently be a high value-add and transformative product to your business. You will want to continue to nurture and develop people to expertly use all the features and be able to bestow that capability with others. The more people using Remote Assist, the more people are exposed to XR in an easy-to-digest way. That is vital in your long-term success. You’re not going to be able to implement this new communication method with an isolated group of skilled operators.

Extended reality is here to stay! There is no doubt about it. Believe me – if you don’t get on board now, you’ll be ten years behind in just three years.

I welcome you to connect with me on Twitter and share your experiences! I love hearing what others are doing on the implementation side of things. @TheNerdNick #doSomethingGreatToday #XR

[Guest Blog] XR and How To Bridge The Gap

by Contributed | Feb 25, 2021 | Technology

This article is contributed. See the original author and article here.

This article was written by Nick Hughes, an avid contributor in the extended reality space as part of our Humans of Mixed Reality Guest Blogger Series. Nick Hughes shares about his approach and core best practices to the Mixed Reality stack and how YOU can bring value in XR projects.

The whole extended reality space is vast, fluid, and sometimes foolish to try to define. There are so many ways that we can leverage this innovative modality that it also makes it somewhat difficult to pitch. The idea of bridging the gap between extended reality and the tangible world is still relatively fresh to most people. In this article, I’m going to give my experience taking a pragmatic approach to the Microsoft mixed reality stack and how you can bring value to your XR projects.

The first ever HoloLens I used is still in my office!

You’re probably thinking to yourself, “This guy is probably a Microsoft partner or some industry expert in mixed reality.” I assure you that two years ago I was running network cables down soot-covered walls in a foundry. 

If there is someone who has asked all the silly questions, someone who has had to search the acronym on the Internet, it is me. However, that is why what I am going to share might be helpful to you too. I am a jack-of-all-trades-master-of-none type of person who just happens to be exceptional at translating all things technology into a humanly digestible form. This is going to land right in your backyard.

Two years ago, I was first exposed to XR. I put on a friend’s PlayStation VR headset and found myself captivated with the experience. The next couple of months had me buried in articles, YouTube videos, blogs, you name it. When I found the HoloLens, I couldn’t stop imagining all the business applications and use case scenarios. I knew this was going to be transformative, but I didn’t know it was going to radically change my entire career direction as well.

Today, I help lead an extended reality team that covers the entire globe. We have deployed HoloLens and Remote Assist in nearly twenty countries and on almost every continent. This technology has dramatically transformed the way we can conduct business and I’m thrilled to have the opportunity to share with everyone how we did it.

1. SHARE THIS AS SOON AS YOU CAN
   I cannot stress to you the importance of excitement. Gordon B. Hinckley said, “Every good citizen adds to the strength of a nation” and that should be your mindset when you first get your hands on a HoloLens. Your first goal should be to create as many “good citizens” of XR as possible. These people will help you to develop the organic business excitement that will be instrumental in the adoption, peer-aided adoption, and funding – yes funding! – of your efforts.
   A word of caution though, if you post all these showy videos in public places, once that excitement gets to a certain point, you’ll have to deliver that capability. Try to be intentional and align your program sponsors before it gets to that level. They may want you to temper that excitement and keep it regulated.
   
   


2. BECOME A PART OF THE MIXED REALITY COMMUNITY
   If you’ve found us here on the Mixed Reality Tech Community, you’re off to a good start! I and most ordinary humans can only partially consume all the information that is whirring around about XR. Developers are releasing original ideas and capabilities each day. A couple of suggestions that I recommend are the Microsoft Mixed Reality Community, Twitter, and the Global XR Community. These have been very effective in building my XR wisdom and helping flatten my learning curve.
   
   Don’t be intimidated – it is going to seem foreign to you at first and that’s OK. As my friend, Steve Kaminski over at Microsoft would say, “just hang on by your fingernails to what you don’t know at first. If you stay engaged, it will all begin to make sense soon”. If some of the content doesn’t hit home, go jump into AltSpaceVR for a support group or post something in the mixed reality forums. The XR community is wonderfully accommodating and we’re all here to learn together.
   
   


3. REMOTE ASSIST, TRAINING, INTEGRATION – SAY IT WITH ME
   The most straightforward value proposition is the cost savings from eliminating travel altogether. You will crush geographical barriers by traversing over border closings and eliminating the impact of COVID travel bans with ease. This will be huge for anyone doing business globally. All three of those elements can compound into a trifecta business case that is impossible to ignore. 
   Case in point: We are getting complete returns on our Microsoft stack investments in less than six months (~ 3 months) of having a device in operation. It practically pays for itself over time! Now THAT is powerful.
   
   Training is another significant piece of business that is challenging with the current health concerns. Remote Assist and Guides both facilitate the pseudo-face-to-face training as well as virtually guided training. This eliminates the need for another person to be present and helps mitigate risk as well as reduces labor cost. The icing on the cake is that I can normally train a slow adopter of technology how to use Guides in less than an hour because it is so simple. Two weeks into our rollout, a senior manager had come into an MTC meeting and asked “What is the next big thing? What does the future look like?” I can confidently answer that with a single word, integration. It isn’t a new technology that is next, but how we use extended reality. Connect your MES systems, connect your ERP and show your “good citizens” how to advance customized applications. That is like opening Pandora’s box.
   
   


4. CAPTURE THE STORY AND MAKE IT REAL
   Microsoft, specifically, has done a great job at making sure they have enough videos and knowledge available so that some of the stories can be told without too much trouble. However, to get that essential excitement, you’re going to want to show you and your people using XR at your own locations and in your own ways. So, let me accelerate that effort for you.
   As soon as you get your HoloLens 2, grab a laptop on the same network, go to the device portal, and then to the capture section. Now, open your HoloLens and record from the HoloLens. You can capture a brilliant first-person perspective of what you’re viewing. This will be meaningful when you’re making your marketing materials, especially for your executive stakeholders.
   
   You will also want to go buy a Microsoft Display Adapter. I know, I’m sounding like a bit of a salesman for Microsoft (trust me, I wish I got commissions too!) but, this adapter is small, inexpensive, and works without any configuration. Open the box, stick it in the HDMI and USB, and magically stream the first-person video directly to the television. This comes in very convenient when you’re showing someone the HoloLens for the first time!
   
   


5. KEEP ORBITING
   If you take any advice from me it is this, “stay present and keep orbiting”. This technology will sell itself but you must keep in touch and you must be ready to seize an opportunity when it presents itself. You’ll be working quietly for a week and then someone will tell you about this new thing they want to explore! Yes! That is exactly what you want to happen.
   
   Carpe diem, friends!
   
   Do not let those kinds of opportunities go to waste because those “good citizens” are realizing the additional capability to the business. Be a facilitator and help them develop their ideas. That will resonate back to the business in a grand way. Especially when the users can build and then share their own story. You’ll see dividends from the peer-aided adoption that will come from those interactions.
   
   Remote Assist will consistently be a high value-add and transformative product to your business. You will want to continue to nurture and develop people to expertly use all the features and be able to bestow that capability with others. The more people using Remote Assist, the more people are exposed to XR in an easy-to-digest way. That is vital in your long-term success. You’re not going to be able to implement this new communication method with an isolated group of skilled operators.

Extended reality is here to stay! There is no doubt about it. Believe me – if you don’t get on board now, you’ll be ten years behind in just three years.

I welcome you to connect with me on Twitter and share your experiences! I love hearing what others are doing on the implementation side of things. @TheNerdNick #doSomethingGreatToday #XR

Sync Up – a OneDrive podcast : Episode 18, “Better together: Teams+ OneDrive”

by Contributed | Feb 25, 2021 | Technology

This article is contributed. See the original author and article here.

Sync Up is your monthly podcast hosted by the OneDrive team taking you behind the scenes of OneDrive, shedding light on how OneDrive connects you to all your files in Microsoft 365 so you can share and work together from anywhere. You will hear from experts behind the design and development of OneDrive, as well as customers and Microsoft MVPs. Each episode will also give you news and announcements, special topics of discussion, and best practices for your OneDrive experience.

So, get your ears ready and Subscribe to Sync up podcast!

Meet your show hosts and guests for the episode:

 

   

 

 

 

 

 

 

Jason Moore is the Principal Group Program Manager for OneDrive and the Microsoft 365 files experience.  He loves files, folders, and metadata. Twitter: @jasmo 

Ankita Kirti is a Product Manager on the Microsoft 365 product marketing team responsible for OneDrive for Business. Twitter: @Ankita_Kirti21

 

Cory Kincaid  is a Customer Success Manager for Modern Work, who advises customers on how to use technologies like Teams and OneDrive to improve their business productivity.

 

Additional guests: 

 

Ryan Voelki and Tatyanah Castillo, also customer success managers for Modern Work, who take a #HumansFirst approach to helping customers navigate their digital transformation.

 

Quick links to the podcast

• Main show page


• Apple Podcasts


• RSS


• Stitcher


• Spotify


• Google


• TuneIn


• Listen and subscribe to other Microsoft podcasts at aka.ms/microsoft/podcasts

 

Links to resources mentioned in the show:

• What’s new in Microsoft Teams aka.ms/teamsresources


• Do it on Teams [training] aka.ms/doitonteams


• Exploring the Science of Work and Ingenuity: WorkLab: Vital facts about the future of work. (microsoft.com)


• Welcome to Microsoft Teams | Microsoft Docs


• Creating customized meetings experiences with apps in Microsoft Teams – Microsoft Tech Community


• Tips for Teams meetings (Video)


• OneDrive Tech Community Blog 


• OneDrive On-Demand  – a one stop shop for all the podcast, webinar and customer evidence related to OneDrive.


• Microsoft Docs – The home for Microsoft documentation for end users, developers, and IT professionals.   


• Microsoft Tech Community Home 


• M365 Roadmap


• Microsoft Ignite the tour


• Stay on top of Office 365 changes
  

   

  
  

Be sure to visit our show page to hear all the episodes, access the show notes, and get bonus content. And stay connected to the OneDrive community blog where we’ll share more information per episode, guest insights, and take any questions from our listeners and OneDrive users. We, too, welcome your ideas for future episodes topics and segments. Keep the discussion going in comments below.

 

As you can see, we continue to evolve OneDrive as a place to access, share, and collaborate on all your files in Office 365, keeping them protected and readily accessible on all your devices, anywhere. We, at OneDrive, will shine a recurring light on the importance of you, the user.  We will continue working to make OneDrive and related apps more approachable. The OneDrive team wants you to unleash your creativity. And we will do this, together, one episode at a time.

 

Thanks for your time reading and listening to all things OneDrive,

Ankita Kirti – OneDrive | Microsoft

How to build an FAQ bot for Microsoft Teams with Power Virtual Agents

by Contributed | Feb 25, 2021 | Technology

This article is contributed. See the original author and article here.

In this blog I want to show you, how you can build, test and publish an FAQ bot for Microsoft Teams within minutes. We will use the Power Virtual Agents for Teams, which means, that you will not need any additional license to your Microsoft 365 license, for reference see also Power Virtual Agents for Microsoft Teams plan.

What is Power Virtual Agents?

Power Virtual Agents belongs like Power Apps, Power Automate and Power Bi to the Power Platform (wow, that was a powerFULL sentence :smiling_face_with_halo:). You can create chatbots, which can interact with users in apps and websites, trigger workflows and more, without the need of writing code. You can choose if you want to use it in the Power Virtual Agents standalone web app or as app within Microsoft Teams.

Let’s build a bot

I will guide you how to create an FAQ bot. To feed our bot we will need some FAQ so that our can bot can learn them. I will use FAQ regarding licensing :nerd_face:, but you can choose any FAQ from a website or PDF or even Word file that you like.

• Open Teams


• Click on the Apps icon


• Search for Power Virtual Agents

• Click Add

• Select the Team you want your bot to join

• Give your bot a name ans select a language that your bot shall understand (should be the same language as your FAQ)

• Click Chatbots – here you get an overview of ALL your chatbots

Add topics from any website

• Click Topics

You see, that some basic topics are already created for you. You can take a look later.

• Click Suggested

Now we want to work on feeding our bot with the FAQ from the website that we selected.

• Copy the URL auf the FAQ website


• Paste the URL into the Link to online content field

• Click Add


• Click Start

This may take now a couple of minutes. Grab a coffee in the meanwhile: :hot_beverage:. Soon you will see the message that your new suggested topics are now in:

Review & edit topics

You can now review and edit each topic:

• Click Save Topic

After you are done with reviewing and editing your topics, you will need to turn on the topics

• Switch the toggle to on

Train your bot by entering more trigger phrases. This way, it is more likely that the Chatbot understands users asking questions even if they don’t exactly match the trigger phrases.

Time to test the bot!

You can now review and edit your topics until you are happy with the results.

Publish your Bot to Microsoft Teams

• Click the Publish icon

• Click Add

• Click Add to Teams

• Use your bot

Conclusion & what’s next

It took us only a few minutes to create, test and publish a chatbot, that now works inside of Microsoft Teams. Want to do some more? We could extend the capabilities of our Power Virtual Agents bot: Let’s say our bot can’t answer a question and needs to transfer the chat to a human agent, who will answer that question. What if we trained the bot with that answer so that our bot gets smarter over time? I will cover that in one of my next blog posts. What do you use chatbots for? Did you already try to make a 5 minute bot? Please share below :)

Ombromanie: Creating Hand Shadow stories with Azure Speech and TensorFlow.js Handposes

by Contributed | Feb 25, 2021 | Technology

This article is contributed. See the original author and article here.

Have you ever tried to cast hand shadows on a wall? It is the easiest thing in the world, and yet to do it well requires practice and just the right setup. To cultivate your #cottagecore aesthetic, try going into a completely dark room with just one lit candle, and casting hand shadows on a plain wall. The effect is startlingly dramatic. What fun!

Even a tea light suffices to create a great effect

In 2020, and now into 2021, many folks are reverting back to basics as they look around their houses, reopening dusty corners of attics and basements and remembering the simple crafts that they used to love. Papermaking, anyone? All you need is a few tools and torn up, recycled paper. Pressing flowers? All you need is newspaper, some heavy books, and patience. And hand shadows? Just a candle.

This TikTok creator has thousands of views for their handshadow tutorials

But what’s a developer to do when trying to capture that #cottagecore vibe in a web app?

High Tech for the Cottage

While exploring the art of hand shadows, I wondered whether some of the recent work I had done for body poses might be applicable to hand poses. What if you could tell a story on the web using your hands, and somehow save a video of the show and the narrative behind it, and send it to someone special? In lockdown, what could be more amusing than sharing shadow stories between friends or relatives, all virtually?

Hand shadow casting is a folk art probably originating in China; if you go to tea houses with stage shows, you might be lucky enough to view one like this!

A Show Of Hands

When you start researching hand poses, it’s striking how much content there is on the web on the topic. There has been work since at least 2014 on creating fully articulated hands within the research, simulation, and gaming sphere:

MSR throwing hands

There are dozens of handpose libraries already on GitHub:

• An entire GitHub topic on hand tracking


• ‘Awesome’ list for hand tracking


• Challenges and hackathons

There are many applications where tracking hands is a useful activity:

• Gaming
• Simulations / Training
• “Hands free” uses for remote interactions with things by moving the body
• Assistive technologies
• TikTok effects :trophy:
• Useful things like Accordion Hands apps

One of the more interesting new libraries, handsfree.js, offers an excellent array of demos in its effort to move to a hands free web experience:

Handsfree.js, a very promising project

As it turns out, hands are pretty complicated things. They each include 21 keypoints (vs PoseNet’s 17 keypoints for an entire body). Building a model to support inference for such a complicated grouping of keypoints has provenn challenging.

There are two main libraries available to the web developer when incorporating hand poses into an app: TensorFlow.js’s handposes, and MediaPipe’s. HandsFree.js uses both, to the extent that they expose APIs. As it turns out, neither TensorFlow.js nor MediaPipe’s handposes are perfect for our project. We will have to compromise.

• 
  

  TensorFlow.js’s handposes allow access to each hand keypoint and the ability to draw the hand to canvas as desired. HOWEVER, it only currently supports single hand poses, which is not optimal for good hand shadow shows.

  
  


• 
  

  MediaPipe’s handpose models (which are used by TensorFlow.js) do allow for dual hands BUT its API does not allow for much styling of the keypoints so that drawing shadows using it is not obvious.

  
  

One other library, fingerposes, is optimized for finger spelling in a sign language context and is worth a look.

Since it’s more important to use the Canvas API to draw custom shadows, we are obliged to use TensorFlow.js, hoping that either it will soon support multiple hands OR handsfree.js helps push the envelope to expose a more styleable hand.

Let’s get to work to build this app.

Scaffold a Static Web App

As a Vue.js developer, I always use the Vue CLI to scaffold an app using vue create my-app and creating a standard app. I set up a basic app with two routes: Home and Show. Since this is going to be deployed as an Azure Static Web App, I follow my standard practice of including my app files in a folder named app and creating an api folder to include an Azure function to store a key (more on this in a minute).

In my package.json file, I import the important packages for using TensorFlow.js and the Cognitive Services Speech SDK in this app. Note that TensorFlow.js has divided its imports into individual packages:

“@tensorflow-models/handpose“: “^0.0.6“,

“@tensorflow/tfjs“: “^2.7.0“,

“@tensorflow/tfjs-backend-cpu“: “^2.7.0“,

“@tensorflow/tfjs-backend-webgl“: “^2.7.0“,

“@tensorflow/tfjs-converter“: “^2.7.0“,

“@tensorflow/tfjs-core“: “^2.7.0“,

…

“microsoft-cognitiveservices-speech-sdk“: “^1.15.0“,

Set up the View

We will draw an image of a hand, as detected by TensorFlow.js, onto a canvas, superimposed onto a video suppled by a webcam. In addition, we will redraw the hand to a second canvas (shadowCanvas), styled like shadows:

<div id=“canvas-wrapper column is-half”>

<canvas id=“output” ref=“output”></canvas>

    <video

        id=“video”

        ref=“video”

        playsinline

        style=”

          -webkit-transform: scaleX(-1);

           transform: scaleX(-1);

           visibility: hidden;

           width: auto;

           height: auto;

           position: absolute;

         “

    ></video>

 </div>

 <div class=“column is-half”>

    <canvas

       class=“has-background-black-bis”

       id=“shadowCanvas”

       ref=“shadowCanvas”

     >

    </canvas>

</div>

Load the Model, Start Keyframe Input

Working asynchronously, load the Handpose model. Once the backend is setup and the model is loaded, load the video via the webcam, and start watching the video’s keyframes for hand poses. It’s important at these steps to ensure error handling in case the model fails to load or there’s no webcam available.

async mounted() {

    await tf.setBackend(this.backend);

    //async load model, then load video, then pass it to start landmarking

    this.model = await handpose.load();

    this.message = “Model is loaded! Now loading video“;

    let webcam;

    try {

      webcam = await this.loadVideo();

    } catch (e) {

      this.message = e.message;

      throw e;

    }

    this.landmarksRealTime(webcam);

  },

Setup the Webcam

Still working asynchronously, set up the camera to provide a stream of images

async setupCamera() {

      if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {

        throw new Error(

          “Browser API navigator.mediaDevices.getUserMedia not available“

        );

      }

      this.video = this.$refs.video;

      const stream = await navigator.mediaDevices.getUserMedia({

        video: {

          facingMode: “user“,

          width: VIDEO_WIDTH,

          height: VIDEO_HEIGHT,

        },

      });

      return new Promise((resolve) => {

        this.video.srcObject = stream;

        this.video.onloadedmetadata = () => {

          resolve(this.video);

        };

      });

    },

Design a Hand to Mirror the Webcam’s

Now the fun begins, as you can get creative in drawing the hand on top of the video. This landmarking function runs on every keyframe, watching for a hand to be detected and drawing lines onto the canvas – red on top of the video, and black on top of the shadowCanvas. Since the shadowCanvas background is white, the hand is drawn as white as well and the viewer only sees the offset shadow, in fuzzy black with rounded corners. The effect is rather spooky!

async landmarksRealTime(video) {

      //start showing landmarks

      this.videoWidth = video.videoWidth;

      this.videoHeight = video.videoHeight;

      //set up skeleton canvas

      this.canvas = this.$refs.output;

      …
      //set up shadowCanvas

      this.shadowCanvas = this.$refs.shadowCanvas;

      …
      this.ctx = this.canvas.getContext(“2d“);

      this.sctx = this.shadowCanvas.getContext(“2d“);
      …
      //paint to main
      this.ctx.clearRect(0, 0, this.videoWidth,

  this.videoHeight);

      this.ctx.strokeStyle = “red“;

      this.ctx.fillStyle = “red“;

      this.ctx.translate(this.shadowCanvas.width, 0);

      this.ctx.scale(–1, 1);
      //paint to shadow box
      this.sctx.clearRect(0, 0, this.videoWidth, this.videoHeight);

      this.sctx.shadowColor = “black“;

      this.sctx.shadowBlur = 20;

      this.sctx.shadowOffsetX = 150;

      this.sctx.shadowOffsetY = 150;

      this.sctx.lineWidth = 20;

      this.sctx.lineCap = “round“;

      this.sctx.fillStyle = “white“;

      this.sctx.strokeStyle = “white“;
      this.sctx.translate(this.shadowCanvas.width, 0);

      this.sctx.scale(–1, 1);
      //now you’ve set up the canvases, now you can frame its landmarks

      this.frameLandmarks();

    },

For Each Frame, Draw Keypoints

As the keyframes progress, the model predict new keypoints for each of the hand’s elements, and both canvases are cleared and redrawn.

      const predictions = await this.model.estimateHands(this.video);

      if (predictions.length > 0) {

        const result = predictions[0].landmarks;

        this.drawKeypoints(

          this.ctx,

          this.sctx,

          result,

          predictions[0].annotations

        );

      }

      requestAnimationFrame(this.frameLandmarks);

Draw a Lifelike Hand

Since TensorFlow.js allows you direct access to the keypoints of the hand and the hand’s coordinates, you can manipulate them to draw a more lifelike hand. Thus we can redraw the palm to be a polygon, rather than resembling a garden rake with points culminating in the wrist.

Re-identify the fingers and palm:

     fingerLookupIndices: {

        thumb: [0, 1, 2, 3, 4],

        indexFinger: [0, 5, 6, 7, 8],

        middleFinger: [0, 9, 10, 11, 12],

        ringFinger: [0, 13, 14, 15, 16],

        pinky: [0, 17, 18, 19, 20],

      },

      palmLookupIndices: {

        palm: [0, 1, 5, 9, 13, 17, 0, 1],

      },

…and draw them to screen:

    const fingers = Object.keys(this.fingerLookupIndices);

      for (let i = 0; i < fingers.length; i++) {

        const finger = fingers[i];

        const points = this.fingerLookupIndices[finger].map(

          (idx) => keypoints[idx]

        );

        this.drawPath(ctx, sctx, points, false);

      }

      const palmArea = Object.keys(this.palmLookupIndices);

      for (let i = 0; i < palmArea.length; i++) {

        const palm = palmArea[i];

        const points = this.palmLookupIndices[palm].map(

          (idx) => keypoints[idx]

        );

        this.drawPath(ctx, sctx, points, true);

      }

With the models and video loaded, keyframes tracked, and hands and shadows drawn to canvas, we can implement a speech-to-text SDK so that you can narrate and save your shadow story.

To do this, get a key from the Azure portal for Speech Services by creating a Service:

You can connect to this service by importing the sdk:

import * as sdk from “microsoft-cognitiveservices-speech-sdk”;

…and start audio transcription after obtaining an API key which is stored in an Azure function in the /api folder. This function gets the key stored in the Azure portal in the Azure Static Web App where the app is hosted.

async startAudioTranscription() {

      try {

        //get the key

        const response = await axios.get(“/api/getKey“);

        this.subKey = response.data;

        //sdk

        let speechConfig = sdk.SpeechConfig.fromSubscription(

          this.subKey,

          “eastus“

        );

        let audioConfig = sdk.AudioConfig.fromDefaultMicrophoneInput();

        this.recognizer = new sdk.SpeechRecognizer(speechConfig, audioConfig);
        this.recognizer.recognized = (s, e) => {

          this.text = e.result.text;

          this.story.push(this.text);

        };
        this.recognizer.startContinuousRecognitionAsync();

      } catch (error) {

        this.message = error;

      }

    },

In this function, the SpeechRecognizer gathers text in chunks that it recognizes and organizes into sentences. That text is printed into a message string and displayed on the front end.

Display the Story

In this last part, the output cast onto the shadowCanvas is saved as a stream and recorded using the MediaRecorder API:

const stream = this.shadowCanvas.captureStream(60); // 60 FPS recording

      this.recorder = new MediaRecorder(stream, {

        mimeType: “video/webm;codecs=vp9“,

      });

      (this.recorder.ondataavailable = (e) => {

        this.chunks.push(e.data);

      }),

        this.recorder.start(500);

…and displayed below as a video with the storyline in a new div:

      const video = document.createElement(“video“);

      const fullBlob = new Blob(this.chunks);

      const downloadUrl = window.URL.createObjectURL(fullBlob);

      video.src = downloadUrl;

      document.getElementById(“story“).appendChild(video);

      video.autoplay = true;

      video.controls = true;

This app can be deployed as an Azure Static Web App using the excellent Azure plugin for Visual Studio Code. And once it’s live, you can tell durable shadow stories!

Try Ombromanie here. The codebase is available here

Take a look at Ombromanie in action:

Learn more about AI on Azure
Azure AI Essentials Video covering speech and language
Azure free account sign-up