Microsoft Project and Dynamics 365 Project Operations | Updates for 2021

by Contributed | Jun 1, 2021 | Technology

This article is contributed. See the original author and article here.

Take a look at new project management options from Microsoft, and how they meet your project needs. From integrated experiences with Microsoft Teams using Project for the web, all the way to Project Operations in Dynamics 365 to support the delivery of project-based services with seamless integration into your ERP system, and manage all phases of your projects.

We’ve set out to take our mature project management engine with Microsoft Project, and make project management more accessible to everyone in your organization. George Bullock, joins host Jeremy Chapman, to share project management options for sophisticated projects.

Project for the web

Now use it within the context of Microsoft Teams. Easy to create and manage projects — see completed tasks, track progress, and modify details. Contacts are in Teams, natively, right in your directory service.

Dynamics 365 Project Operations

Project for the web capabilities are embedded within, so core project planning and resource management is the same. Expanded set of capabilities across sales management and project accounting. Easily manage your entire project life cycle, from planning to billing and finance.

QUICK LINKS: 

02:32 — Project for the web in Teams 

04:30 — How to manage a more sophisticated project 

06:06 — Project Operations in Dynamics 365 

08:46 — Billing capabilities 

09:33 — Finance capabilities 

09:59 — Set up: Project for the web 

10:47 — Set up: Project Operations 

11:27 — Wrap up

Link References: 

For more on how to set up Project for the web, go to https://aka.ms/ProjectMechanics 

Request a trial for Dynamics 365 Project Operations at https://aka.ms/ProjectOperationsTrial 

Get a Project trial at aka.ms/TryProjectNow

Unfamiliar with Microsoft Mechanics?

We are Microsoft’s official video series for IT. You can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. 

• Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries?sub_confirmation=1 


• Join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog 


• Watch or listen via podcast here: https://microsoftmechanics.libsyn.com/website 

Keep getting this insider knowledge, join us on social: 

• Follow us on Twitter: https://twitter.com/MSFTMechanics 


• Follow us on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ 


• Follow us on Facebook: https://facebook.com/microsoftmechanics/

Video Transcript:

– Up next, we look at project management options from Microsoft and how they meet your project needs, from integrated experiences with Microsoft Teams using Project for the web all the way to Dynamics 365’s Project Operations to support the delivery of project-based services with seamless integration into your ERP system to manage all phases of your projects from bidding to billing. So, George, it’s great to have you on.

– Well, thanks for having me.

– And thanks so much for joining us from home today. So one of the most popular topics for Microsoft Mechanics has always been around project management. So Microsoft has this really long history of building tools that span from task management and planning, formal project management, and even business operations. And today though, we wanted to focus more on that sophisticated end of the spectrum. So what’s new there?

– Well, we’ve really focused on making it easier for everyone to manage their work. And if you think about it, most of us are involved in projects every day. They can range from single initiatives that you may be collaborating on with your team, such as this video, actually. Or they can span an entire portfolio of interrelated initiatives and business processes across your organization. And also, there are project-based services that might be a primary driver for your business, especially if you’re in a business like consulting, where you’re managing everything from sales to running the project, as well as post-project billing and analytics.

– Right, and really the more accountability and dependencies involved, the more rigor that you need to be able to follow those projects through.

– Yeah, that’s exactly right. And we’ve set out to take our mature project management engine with Microsoft Project, and make the whole discipline of project management way more accessible to everyone in your organization. For example, Project for the web has been around for a couple of years now and can be used as your go-to tool if project management is your whole job focus. The good news is as an everyday user, you can also now use it within the context of Microsoft Teams. Project for the web and its rich scheduling and task management capabilities are also embedded in Dynamics 365 Project Operations. And by the way, this is the evolution of Dynamics 365 Project Service Automation. It gives you an expanded set of capabilities across sales management and project accounting for Project Operations and starts with sales and quotes, and goes through to planning and running the project and all the way to things like expenses, invoices, and analytics.

– Right, and there’s also good news for IT that it’s built on top of the Power Platform. So you’ve had to deploy things like project servers in the past, all that pain’s going to go away and we’re covering both ends of the spectrum there then for work management. But can you give us a look of all the new tools?

– Sure. I’ll start with Project for the web in Microsoft Teams to show you how easy it is to create and manage projects. The first thing I’m going to do here is hit the plus sign to add a tab and I’ll add project. And you see now when I click into project I can either add an existing project to my team, or I can create a new one. I’ll go ahead and create a new one and give it a name, Marketing Plan, and click Save. And now this project is being attached to our channel and everyone on the team will be able to see it with access to the project. From here, we can start adding tasks. So I’ll add a few for email campaign, audience profile and storyboard. And from here I can also create a work breakdown structure. And in this case tasks 2 and 3, we’ll make those subtasks to our email campaign. And now I can begin assigning people; I’ll assign Adele for now.

– Right, and this is really a big deal because in the past, you’d have to manage another contact list of people that were defined as project resources, and now they’re just there natively in your directory service.

– That’s right, it’s a huge time saver for sure. Also the cool thing is if there’s someone that needs to be on this project that’s in our tenant, but not yet a member of this actual team, I can search for them. So here I’ll search for Brian Johnson and add him to the project. And I can also choose to add him to the team in Microsoft Teams, so he has the full experience including associated chat and files. And so now I have everyone in and I can lay out the project schedule. And you’ll notice there’s a Duration column here by default. So I’ll assign each task the number of days required. I’ll add start and finish date columns, and you can see the project uses today’s date as the default start date. Then behind the scenes, the scheduling engine will automatically calculate the finish date for each task. And now from here, I can add additional fields, including custom fields. And as I add a column, I can choose the data type. In this case I’ll choose Yes/No, and I’ll give this field the name Legal Review and hit Create. And now I can assign yes or no to each of my tasks. And that’s how easy it is to set up your project.

– Okay, but I have to say that was a pretty simple project, but can you show us what a more sophisticated project then might look like?

– Yeah, absolutely. Here I have a project for Cafe Grande 2000, and this one’s further along. So you can easily see completed tasks with green check marks and strikethrough texts. The tasks that are behind schedule are highlighted in red and tasks due tomorrow are highlighted in beige. So I’ll mark a few of these tasks complete. Now, if you look at this one here for Identify online opportunities, you can see that it has a dependency on defined launch goals, and I can easily modify all the details of this task from here. So now I’ll close this out. And of course, one of the most visual ways to track progress and view dependencies is through the timeline. And this is a great view that shows all the project tasks, how they’re related to each other, and when they’ll be completed. If I want, I can modify each task on the timeline by sliding them to the left or right, and I can shorten or extend their duration as well. Now, if I click into any of these tasks on the left, it gives me the details behind them, as you can see with this one. So let’s close this out and go to another useful view that we have here, which is the Board view. And this lets you categorize tasks by buckets. And it’s really easy to drag tasks between buckets. So I’ll drag a few of these tasks over into their corresponding buckets. Now we get a better idea in this case of the phases each task belongs to.

– So this is going to look pretty familiar, I think, to a lot of people in project management. Even you have no prior experience, it’s going to be easy to onboard, and the different views are going to give you options then for tracking their progress. This is great for managing everyday products, but you also mentioned before how we go beyond this with Project Operations in Dynamics 365, especially if you’re selling and running project-based services, but what does that experience then look like?

– Yeah, let me show you that. And the first thing you’ll notice is that because Project for the web capabilities are embedded within Dynamics 365 Project Operations, the core project planning and resource management is the same as Project for the web. But Project Operations goes far beyond this. It comes with a suite of capabilities that can help project leads, account managers, and project accountants more easily manage the project life cycle. So starting with deal management, Project Operations has robust estimation capabilities. As you can see here, the team for this project uses generic resources, so project managers can create accurate estimates. Account managers can then take these estimates and link them directly to project quotes, helping them create far more accurate quotes for their customers. Clicking on the Sales tab now, we can easily find the link quote. Now in the Profitability Analysis tab, account managers can even use these estimates to identify the potential gross margin for the project.

– Okay, so then once you’re ready to move forward with the project, how does Project Operations help you then assign the resources to carry out all of your planning?

– Well, project leads can now use the Schedule Board to easily see who is active on other projects, and if people with the needed skillsets have available time to staff the new project. So in this view, project leads can search by role, skillset, location, and a number of other valuable filters. And for this new project, they’ll need a site foreman who has a lot of project management experience. So after searching, we can see that Peter has plenty of free time to devote to this new project in the upcoming months. Then once you’ve selected your team and the project is kicked off, you have workflows in place to simplify time and expense management. In Time Entries, project team members can quickly add their time sheets by importing their hours from their current project or by simply manually adding them in. And for Expenses, we have two options. First, team members can submit their expenses using this simple experience here. If you want more advanced capabilities like OCR for receipt capture, we have an option for that as well to save time when submitting these expenses. Project Operations also integrates ERP capabilities for things like material sourcing. So here, for example, this project team is working to refurbish and rebrand a number of cafes. They’ve identified that they need to equip these cafes with all new equipment, like new coffee machines with material usage. So I’ll go ahead and add our A100 model in this case. And they can easily make sure the expense is accounted for and eventually invoiced to the customer.

– Okay, and I know that it can also be a challenge once you move into the billing aspects of a project, so what billing capabilities then come with Project Operations?

– Yeah, so we’ve just introduced a new feature, actually, that lets project leads define whether certain tasks of a project should be complimentary or not chargeable to the customer, like a proof of concept or pre-sales activities, while the remaining tasks may require a fixed fee or time and materials billing. So for this project, we can change just the planning activities to be complimentary. And project leads can even bill multiple organizations for different sets of tasks under the same project. Now switching to the Tracking tab, project leads can use this table you see to easily track the project from both an effort and a cost standpoint, and this helps make sure the project remains under budget.

– From a project accounting perspective, are there some additional finance capabilities here as well?

– Yeah, there are. Project Operations helps accounting teams manage all the financial aspects of a project, from project costing, to invoicing and revenue recognition. With Project Statements, for example, project accountants can monitor the latest project financials and make sure the project’s on the right track. As you can see here with profit and loss and also in this tab looking at consumption.

– Okay, so this is a really great set of integrated project management capabilities across Project for the web and Project Operations, but what do they take then to get set up?

– Let’s start with Project for the web and what I showed in Teams. So once your users are licensed, it’s really easy to set up. Because first, under the covers, we’re using Microsoft Dataverse, which provides a scalable and secure managed data layer as part of the Power Platform. And we provisioned this with a default environment as part of your enterprise Microsoft 365 subscription, so no architecture or servers to manage. Now, optionally, if you want additional compliance and control, or you want to scope usage across different departments, you can set up named environments and deploy Project for the web into those. For more on how to do that setup, you can go to aka.ms/ProjectMechanics.

– Okay, so what about Project Operations, what deployment options then do you have there?

– Well, depending on what you have now, there are a few paths you can take. First, we have a simple Lite deployment option. And then we have Project Operations for resource and non-stocked materials scenarios for when you don’t carry an inventory of materials or resources. And finally, we have Project Operations for stocked and production order scenarios. To help you determine which deployment is best in your case, we’ve created a deployment questionnaire with five questions. Once you’ve answered the questions, it’ll recommend a deployment option for you. And from there, the Begin Setup button will direct you right to the Lifecycle Services to enable it.

– Thanks so much, George, for walking us through all the options for detailed project management, whether you’re working in Teams or across multiple divisions inside your organization, but how can people then learn more?

– The great news is you can try both solutions for yourself. I recommend requesting a trial for Dynamics 365 Project Operations. You can do that at aka.ms/ProjectOperationsTrial. And for Project for the web, you can visit aka.ms/TryProjectNow.

– Awesome stuff, and of course, keep watching Microsoft Mechanics for the latest tech updates. Subscribe if you haven’t already, and we’ll see you next time.

Time Indicators – Average Time taken to remediate resources

by Contributed | Jun 1, 2021 | Technology

This article is contributed. See the original author and article here.

One of the important goals of Azure Security Center is to help you understand your current security situation. Azure Security Center continuously assesses your resources for security issues and flags a resource as healthy or unhealthy accordingly providing you recommendations.

Automation is not new in Azure Security Center, it reduces overhead and can improve your security by ensuring the process steps are done quickly, consistently and according to your predefined requirements. Azure Resource Graph and Azure Security Center’s REST APIs gives you great capability to achieve this goal. We recently added two new time indicator fields to both Microsoft.security/assessments REST API provider and Azure Resource Graph to help you have more visibility into when was a resource first identified as unhealthy/healthy (firstEvaluationDate) and when the resource status changed to healthy/unhealthy (statusChangeDate) with the help of which, you can create automations around Cloud Security Posture Management (CSPM). Learn more about these fields here. The time-indicator fields makes it even more easy for you to automate and build your own playbooks for e.g., an automation my dear friend created recently to notify about new stale resources.

Today I want to introduce you to another automation that uses Time Indicators (firstEvaluationDate and statusChangeDate) to show the average time it needs for your resource owners to remediate a particular security control or recommendation.

This playbook is configured to run every 24hrs to pull the assessment information from the API and exports the assessments identified by Azure security center to a custom log of the Log Analytics workspace

Example Custom log created in the Log analytics workspace 

Since the Log Analytics workspace has the historical data now, the logic app starts by iterating all the healthy assessments of the resources and if the assessment status is found to be “healthy”, it queries the history to check if that resource was ever “unhealthy” and logs the previousStatus date.

previousStatus calculation example

The difference of these dates will be written to an array to calculate the average in the next loop.

The Logic App is using ticks, to calculate the specific timestamp inorder to get the average time. One of the challenges using the built-in date/time functions in Logic App, addToTime() or addSeconds(), is it can only handle as precise as seconds, while ticks provide ten million times more accurate values, which we lose a certain level of precision.

Use of Ticks to calculate timestamp

The output of the HealthyArray variable will be passed to an Aggregation loop to calculate the average time for all those resources whose previousState was unhealthy.

This value is passed to a to a custom log of the Log Analytics workspace which represents the average time taken to remediate the resources to better RCA for changes.  

Example Custom log created in the Log analytics workspace representing average time

The playbook will also calculate the total number of unhealthy resources in the 24hrs period and the top ‘five’ security controls that are unhealthy in your environment and will then compose a new HTML email body and send it to the address(es) that you determine when deploying the automation to your environment.

Example email generated from the automation

Note:

Make sure to manually trigger the playbook once you’ve met all prerequisites. Since this Playbook creates a custom log in the Log Analytics Workspace, for the first few runs the logicapp will error out until there’s data in the workspace for it to pull. Once the custom table gets populated in the workspace, you’ll start receiving the results.

What are you waiting for? Go ahead and deploy this automation from our Azure Security Center Github repository using the provided ARM template and let us know your feedback.

Now that we have given you a good head start on how to use these fields, show us your creativity using the time indicators and don’t forget to let us know how these new fields (firstEvaluationDate and statusChangeDate) are helpful for you.

Acknowledgements:

Thanks to for @Miri Landau, @Ronny Keshet for envisioning this wonderful automation idea.

Using Microsoft Defender for Identity Data to Make Powerful Advanced Hunting Queries

by Contributed | Jun 1, 2021 | Technology

This article is contributed. See the original author and article here.

It’s been a while since we last talked about the events captured by Microsoft Defender for Identity. We last published a blog in August last year and so we thought it would be a good opportunity to give you an update with the latest events you can use to hunt for threats on your domain controllers using advanced hunting in Microsoft 365 Defender.

As a general rule of thumb, all Defender for Identity activities that are available in Microsoft 365 Defender advanced hunting fit into one of four data sets:

• IdentityInfo


• IdentityLogonEvents


• IdentityQueryEvents


• IdentityDirectoryEvents

(Figure 1 – The advanced hunting console available as part of Microsoft 365 Defender)

Learn about accounts in your organization using IdentityInfo

Every effective threat hunting style investigation starts with understanding what users we are protecting, and this is what we are presented with using the IdentityInfo table. Although this data set is not exclusive to Defender for Identity, it does provide comprehensive details for the accounts being utilized in the environment. Using information made available from this data set, you can easily correlate different account attributes such as cloud / on-premises SID, UPN, and object ID.

This table also provides rich account information from Active Directory such as personal details (name, surname, city, country), professional information (job title, department, email address), and other AD attributes (domain, display name).

//Find out what users are disabled.
IdentityInfo
| where IsAccountEnabled == “0”
| summarize arg_max(AccountName,*) by AccountUpn

Correlating information between Defender for Identity and Defender for Endpoint

With Defender for Identity installed in your organization, your on-premises Active Directory identities are protected against advanced threats, and you also have visibility into various logon events. These authentication activities, along with those being captured by Azure AD feeds into IdentityLogonEvents data set, where you can hunt over authentication activities easily.

Defender for Identity activities are covering authentications over Kerberos, LDAP, and NTLM. Each authentication activity provides details such as the account information, the device the authentication activity was performing on, network information (such as the IP and port number), and more.

Harnessing this data, you can easily hunt over abnormal logons during non-working hours, learn more about logon routine in the network, and correlate data with Microsoft Defender for Endpoint.

// Enrich logon events with network activities happening on the device at the same time
IdentityLogonEvents
| where Timestamp > ago(7d)
| project LogonTime = Timestamp, DeviceName, AccountName, Application, LogonType
| join kind=inner (
DeviceNetworkEvents
| where Timestamp > ago(7d)
| project NetworkConnectionTime = Timestamp, DeviceName, AccountName = InitiatingProcessAccountName, InitiatingProcessFileName, InitiatingProcessCommandLine  
) on DeviceName, AccountName
| where LogonTime - NetworkConnectionTime between (-2m .. 2m)

Queries targeting Active Directory objects

With IdentityQueryEvents, you can quickly find out what queries are targeting the domain controller. Queries can happen over the network either naturally by different services or legitimate activities in the network but can often be used by an attacker to perform reconnaissance activities on different objects like users, groups, devices or domains – seeking out those with certain attributes or privileges.

In certain attack vectors, like AS-REP Roasting that we covered in an earlier blog post, the reconnaissance portion often involves the attacker performing enumeration techniques to find users that have the pre-authentication attribute turned on. This can be found easily with a similar query:

// Enrich logon events with network activities happening on the device at the same time

IdentityLogonEvents
| where Timestamp > ago(7d)
| project LogonTime = Timestamp, DeviceName, AccountName, Application, LogonType
| join kind=inner (
DeviceNetworkEvents
| where Timestamp > ago(7d)
| project NetworkConnectionTime = Timestamp, DeviceName, AccountName = InitiatingProcessAccountName, InitiatingProcessFileName, InitiatingProcessCommandLine  
) on DeviceName, AccountName
| where LogonTime - NetworkConnectionTime between (-2m .. 2m)

Track Active Directory changes

Finally, we have the IdentityDirectoryEvents table. In general, this table captures three categories of events on your domain controller:

• Remote code execution.


• Changes to attributes of Active Directory objects, including groups, users, and devices.


• Other activities performed against the directory, such as replication or SMB session enumeration.

Also, starting with Defender for Identity version 2.148, if you configure and collect event ID 4662, Defender for Identity will report which user made the Update Sequence Number (USN) change to various Active Directory object properties. For example, if an account password is changed and event 4662 is enabled, the event will record who changed the password. As a result, this information can be found using advanced hunting.

Here is a sample query you can use:

// Track service creation activities on domain controllers
IdentityDirectoryEvents
| where ActionType == "Service creation"
| extend ServiceName = AdditionalFields["ServiceName"]
| extend ServiceCommand = AdditionalFields["ServiceCommand"]
| project Timestamp, ActionType, Protocol, DC = TargetDeviceName, ServiceName, ServiceCommand, AccountDisplayName, AccountSid, AdditionalFields
| limit 100

As always, please let us know what you think and how we can enhance this capability further. Let us know what you use advanced hunting for in the comments too!

To learn more about advanced hunting in Microsoft 365 Defender and these new enhancements, go to the following links:

• Advanced hunting overview


• Preview features


• Microsoft 365 Defender GitHub community

Plant AI: Student Ambassador Green-A-Thon activity report

by Contributed | Jun 1, 2021 | Technology

This article is contributed. See the original author and article here.

Hello developers :waving_hand:! In this article, we introduce our project “Plant AI :shamrock:” and walk you through our motivation behind building this project, how it could be helpful to the community, the process of building this project, and finally our future plans with this project.

Plant AI :shamrock: is a web application :globe_with_meridians: that helps to easily diagnose diseases in plants from plant images using Machine Learning available on the web. We provide an interface on the website where you can upload images of your plant leaves. Since we focus on plant leaf diseases we can detect the plant’s diseases by seeing an image of the leaves. We also provide users easy ways to treat the diagnosed disease.

As of now, our model supports 38 categories of healthy and unhealthy plant images across species and diseases. See the complete list of supported diseases and species can be found here. If you are want to test out Plant AI, you can use one of these images.

Guess, what? This project is also completely open-sourced:star:, here is the GitHub repo for this project: https://github.com/Rishit-dagli/Greenathon-Plant-AI

The motivation behind building this

Human society needs to increase food production an estimated 70% by 2050 to feed an expected population size that is predicted to be over 9 billion people [1]. Currently, infectious diseases reduce the potential yield by an average of 40% with many farmers in the developing world experiencing yield losses as high as 100%.

The widespread distribution of smartphones among farmers around the world offers the potential of turning smartphones into a valuable tool for diverse communities growing food.

Our motivation with Plant AI is to aid crop growers by turning their smartphones into a diagnosis tool that could substantially increase crop yield and reduce crop failure. We also aim to make this rather easy for crop growers so the tool can be used on a daily basis.

How does this work?

As we highlighted in the previous section, our main target audience with this project is crop growers. We intend for them to use this on a daily basis to diagnose disease from their plant images.

Our application relies on the Machine Learning Model we built to identify plant diseases from images. We first built this Machine Learning model using TensorFlow and Azure Machine Learning to keep track, orchestrate, and perform our experiments in a well-defined manner. A subset of our experiments used to build the current model have also been open-sourced and can be found on the project’s GitHub repo.

We were quite interested in running this Machine Learning model on mobile devices and smartphones to further amplify its use. Using TensorFlow JS to optimize our model allows it to work on the web for devices that are less compute-intensive.

We also optimized this model to work on embedded devices with TensorFlow Lite further expanding the usability of this project and also providing a hosted model API built using TensorFlow Serving and hosted with Azure Container Registry and Azure Container Instances.

We talk about the Machine Learning aspect and our experiments in greater detail in the upcoming sections.

The model in action

To allow plant growers to easily use this Plant AI, we provide a fully functional web app built with React and hosted on Azure Static Web Apps. This web app allows farmers to use the Machine Learning model and identify diseases from plant images all on the web. You can try out this web app at https://www.plant-ai.tech/ and upload a plant image to our model. In case you want to test out the web app we also provide real-life plant images you can use.

We expect most of the traffic and usage of Plant AI from mobile devices, consequently, the Machine Learning model we run through the web app is optimized to run on the client-side.

This also enables us to have blazing fast performance with our ML model. We use this model on the client-side with TensorFlow JS APIs which also allows us to boost performance with a WebGL backend.

Building the Machine Learning Model

Building the Machine Learning Model is a core part of our project. Consequently, we spent quite some time experimenting and building the Machine Learning Model. We had to build a machine learning model that offers acceptable performance and is not too heavy since we want to run the model on low-end devices

Training the model

We trained our model on the Plant Village dataset [2] on about 87,000 (+ augmented images) healthy and unhealthy leaf images. These images were classified into 38 categories based on species and diseases. Here are a couple of images the model was trained on:

We experimented with quite a few architectures and even tried building our own architectures from scratch using Azure Machine Learning to keep track, orchestrate, and perform our experiments in a well-defined manner.

It turned out that transfer learning on top of MobileNet [3] was indeed quite promising for our use case. The model we built gave us the acceptable performance and was close to 12 megabytes in size, not a heavy one. Consequently, we built a model on top of MobileNet using initial weights from MobileNet trained on ImageNet [4].

We also made a subset of our experiments used to train the final model for public use through this project’s GitHub repository.

Running the model on a browser

We applied TensorFlow JS (TFLS) to perform Machine Learning on the client-side on the browser. First, we converted our model to the TFJS format with the TensorFlow JS converter, which allowed us to easily convert our TensorFlow SavedModel to TFJS format. The TensorFlow JS Converter also optimized the model for the web by sharding the weights into 4MB files so that they can be cached by browsers. It also attempts to simplify the model graph itself using Grappler such that the model outputs remain the same. Graph simplifications often include folding together adjacent operations, eliminating common subgraphs, etc.

After the conversion, our TFJS format model has the following files, which are loaded on the web app:

• model.json (the dataflow graph and weight manifest)


• group1-shard*of* (collection of binary weight files)

Once our TFJS model was ready, we wanted to run the TFJS model on browsers. To do so we again made use of the TensorFlow JS Converter that includes an API for loading and executing the model in the browser with TensorFlow JS :rocket:. We were excited to run our model on the client-side since the ability to run deep networks on personal mobile devices improves user experience, offering anytime, anywhere access, with additional benefits for security, privacy, and energy consumption.

Designing the web app

One of our major aims while building Plant AI was to make high-quality disease detection accessible to most crop growers. Thus, we decided to build Plant AI in the form of a web app to make it easily accessible and usable by crop growers.

As mentioned earlier, the design and UX of our project are focused on ease of use and simplicity. The basic frontend of Plant AI contains just a minimal landing page and two other subpages. All pages were designed using custom reusable components, improving the overall performance of the web app and helping to keep the design consistent across the web app.

Building and hosting the web app

Once the UI/UX wireframe was ready and a frontend structure was available for further development, we worked to transform the Static React Application into a Dynamic web app. The idea was to provide an easy and quick navigation experience throughout the web app. For this, we linked the different parts of the website in such a manner that all of them were accessible right from the home page.

Web landing page

Once we can access the models we load them using TFJS converter model loading APIs by making individual HTTP(S) requests for loading the model.json file (the dataflow graph and weight manifest) and the sharded weight file in the mentioned order. This approach allows all of these files to be cached by the browser (and perhaps by additional caching servers on the internet) because the model.json and the weight shards are each smaller than the typical cache file size limit. Thus a model is likely to load more quickly on subsequent occasions.

We first normalize our images that is to convert image pixel values from 0 to 255 to 0 to 1 since our model has a MobileNet backbone. After doing so we resize our image to 244 by 244 pixels using nearest neighbor interpolation though our model works quite well on other dimensions too. After doing so we use the TensorFlow JS APIs and the loaded model to get predictions on plant images.

Hosting the web app we built was made quite easy for us using Azure Static Web Apps. This allowed us to easily set up a CI/ CD Pipeline and Staging slots with GitHub Actions (Azure’s Static Web App Deploy action) to deploy the app to Azure. With Azure Static Web Apps, static assets are separated from a traditional web server and are instead served from points geographically distributed around the world right out of the box for us. This distribution makes serving files much faster as files are physically closer to end users.

Future Ideas

We are always looking for new ideas and addressing bug reports from the community. Our project is completely open-sourced and we are very excited if you have feedback, feature requests, or bug reports apart from the ones we mention here. Please consider contributing to this project by creating an issue or a Pull Request on our GitHub repo!

One of the top ideas we are currently working on is transforming our web app into a progressive web app to allow us to take advantage of features supported by modern browsers like service workers and web app manifests. We are working on this to allow us to support:

• Offline mode


• Improve performance, using service workers


• Platform-specific features, which would allow us to send push notifications and use location data to better help crop growers


• Considerably less bandwidth usage

We are also quite interested in pairing this with existing on-field cameras to make it more useful for crop growers. We are exploring adding accounts and keeping a track of images the users have run on the model. Currently, we do not store any info about the images uploaded. It would be quite useful to track images added by farmers and store information about disease statistics in a designated piece of land on which we could model our suggestions to treat the diseases.

Thank you for reading!

If you find our project useful and want to support us; consider giving a star :star: on the project’s GitHub repo.

Many thanks to Ali Mustufa Shaikh and Jen Looper for helping me to make this better. :)

Citations

[1] Alexandratos, Nikos, and Jelle Bruinsma. “World Agriculture towards 2030/2050: The 2012 Revision.” AgEcon Search, 11 June 2012, doi:10.22004/ag.econ.288998.

[2] Hughes, David P., and Marcel Salathe. “An Open Access Repository of Images on Plant Health to Enable the Development of Mobile Disease Diagnostics.” ArXiv:1511.08060 [Cs], Apr. 2016. arXiv.org, http://arxiv.org/abs/1511.08060.

[3] Howard, Andrew G., et al. “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.” ArXiv:1704.04861 [Cs], Apr. 2017. arXiv.org, http://arxiv.org/abs/1704.04861.

[4] Russakovsky, Olga, et al. “ImageNet Large Scale Visual Recognition Challenge.” ArXiv:1409.0575 [Cs], Jan. 2015. arXiv.org, http://arxiv.org/abs/1409.0575.