by Contributed | Nov 19, 2020 | Azure, Microsoft, Technology
This article is contributed. See the original author and article here.
As many of you may know I run the Glasgow Azure User Group, and over the last three years I’ve had the privilege to hear a good number of excellent speakers deliver their sessions at the group’s meetups. Recently Tricia Sinclair, delivered a brill session going into Azure DevOps Boards. Something I use daily, but don’t even scratch the surface of functionality wise. So, Tricia’s session opened a whole new world to me.
DevOps Boards and Excel
During her session Tricia mentioned there was a plugin for Excel that allowed you to bulk import or modify DevOps Board items, which I had no idea existed!
This plug in is part of the Azure DevOps Office Integration 2019 feature that allows non-developers to interact with Azure DevOps. Now I don’t believe this tool is only for non-developers, I think it can help everyone that needs to interact with DevOps Boards.
To get to work with this feature you need to have Excel 2010 or later installed, the plug, be a member of the project you want to interact with, have the relevant permissions to modify items in the project you want to interact with and of course be using Azure DevOps. For a full list of prerequisites please do check out the official documentation.
Use Excel to create items in Azure DevOps Boards
Using Azure DevOps Boards to keep track of work items, bugs, whatever it might be is an awesome way to create visibility for you and your team, however, it can be a bit tiresome entering a bunch of those things at a time. And being able to bulk load would be handy and this is what the plugin allows you to do!
Once you install the plug in when you open Excel, you have a tab called Team, which allows you to connect with Azure DevOps within Excel.
So, if you have a new Azure DevOps Project and want to bulk import a bunch of tasks for the team to work through you can do that by connecting Excel to the Project and starting the list.
Create a new DevOps List from Excel
You can also do this if you have new tasks or bugs or issues, to add to an existing project. And I will be using this feature in the future to track my work, especially when I’ve been neglecting my duties and not adding my tasks to the team board.
Update items from Excel to your Azure DevOps Board
What about updating items, you can do that as well. Which again allows you to create items, update items, add more information to items. All of that is available.
The best way I have found to bulk update tasks on an existing project is to have a query setup that pulls the items I want to update and then use that to help pull the information into Excel.
Opening up the plug in within Excel, this time I tell it I want to pull the information from the query.
Pull information from DevOps into Excel
Once the information is pulled down I can update the items that I need to, either bulk closing items, or reassigning the items to others.
It’s also a great opportunity to link any items together using the “Links and Attachment” button or even uploading any supporting information for that item.
Add files to your DevOps Items
Speed up those admin tasks
As I said at the top of this blog post I don’t think this is something only for non-developers I think this is a great tool for everyone, if you set aside time at the end of your week to go through and update your Azure DevOps Board this can help speed things up. I’d love to hear if you are using this plugin and if it’s helping to speed up the management of your items or not!
by Contributed | Nov 18, 2020 | Azure, Microsoft, Technology
This article is contributed. See the original author and article here.
MSTICPy, our CyberSec toolset for Jupyter notebooks, has supported VirusTotal lookups since the very earliest days (the earliest days being only around two years ago!). We recently had a contribution to MSTICPy from Andres Ramirez and Juan Infantes at VirusTotal (VT), which provides a new Python module to access the recently-released version 3 of their API.
As well as a Python module, which provides the interface to lookup IoCs via the API, there is also a sample Jupyter notebook demonstrating how to use it.
As a side note, we’re delighted to get this submission, not just because it brings support for the awesome new VirusTotal API, but it is also first substantial contribution to MSTICPy from anyone outside MSTIC (Microsoft Threat Intelligence Center). Big thanks to Juan and Andres!
The two biggest features of the API are:
- the ability to query relationships between indicators
- the ability to visualize these relationships via an interactive network graph.
There is also an easy-to-use Python object interface exposed via the vt_py and vt_graph_api Python libraries (these are both required by the MSTICPy vtlookup3 module). You can read more about the VT Python packages by following the links in the previous sentence.
Why you would want to use VTLookup3
Alerts and incidents in Azure Sentinel will nearly always have a set of entities attached to them. These entities might be things like IP addresses, hosts, file hashes, URLs, etc. This is common to many SOC environments.
Let’s take an example of Microsoft Defender for Endpoint (MDE) alerts ingested into Azure Sentinel.
Note: MDE is the new name for Microsoft Defender Advance Threat Protection (MDATP) so you will probably see both terms kicking around for a while.
If we look at the incident detail and click on the Entities tab we see a list of entities related to the incident.
As part of your SOC triage you will want to check on whether any of these file hashes are known bad and possibly explore other related files, IPs. domains, etc. MSTICPy allows you to do custom correlation with arbitrary external data sources, including VT.
We will use one of the file hashes taken from this incident as we explore VT Lookup capabilities. Although our example below uses a file hash, you could use a URL, domain or IP Address as your starting point.
Getting started
There is a little bit of setup to do before you can get going.
Note: MSTICPy with extras
We’re in the process of moving a lot of the growing list of MSTICPy dependencies into optional installs. These are known as “extras” in the Python setup world. VirusTotal are the first to be “extra”d but we’re not picking on them. It was just, as a new package, it was easiest to start the dependency refactoring with a new library.
Install msticpy with the “vt3” extra
pip install msticpy[vt3]
or just install the vt_py and vt_graph_api packages directly:
pip install vt-py vt-graph-api nest_asyncio
Note: the nest_asyncio package is required for use in notebooks but not if you’re use the vtlookup3 module and VTLookup3 class in Python code.
In the notebook, import the modules:
from msticpy.sectools.vtlookupv3 import VTLookupV3
import nest_asyncio
And create an instance of the VTLookup3 class.
You need to supply your VirusTotal API key when you create the VTLookup3 instance. You can supply this as a string or store it in your msticpyconfig.yaml configuration file.
This code, taken from the notebook will try to find the VT API key in your configuration.
from msticpy.common.provider_settings import get_provider_settings
vt_key = get_provider_settings(“TIProviders”)[“VirusTotal”].args[“AuthKey”]
Note: in the configuration file you can specify that the API key value be retrieved from an environment variable or from an Azure Key Vault. See MSTICPy Package Configuration for more details.
Using VTLookup3
VTLookup3 has the following methods:
We’ll look at each of these in turn.
Note: we use the terms IoC (indicator of compromise) and observable interchangably in this article. They indicate data items such as IP addresses, URLs, file hashes etc. that may be observed during an attack and thus become indicators of compromise.
Looking up a single IoC/observable
lookup_ioc works much like the other TI providers in MSTICPy except that it returns the results as a pandas DataFrame, even for a single IoC query.
If an entry matching this ID is found in VT it is returned with some basic attributes such as submission times, name, and type.
Since the time stamp is returned as a Unix serial timestamp you might want to re-format into a more readable form using code like this.
example_attribute_df.assign(
first_submission=pd.to_datetime(example_attribute_df.first_submission_date, unit=“s”, utc=True),
last_submission=pd.to_datetime(example_attribute_df.last_submission_date, unit=“s”, utc=True)
)
This adds two columns to the data that display as human-readable datetimes
Note: we plan to change this soon so that the API will return datetime types directly.
We’ve added a convenience function to the notebook that allows you to return full details for a VirusTotal object. This isn’t done by default in the built-in methods but you can use this to query additional data for specific IDs.
Note: get_object will be exposed in the core VTLookup3 interface in a future version. This simply implements the vt_py client.get_object call. You can use the vt_py method directly or go the VirusTotal site to see the details rendered in a more consumable fashion.
Looking up IoC Relationships
The lookup_ioc_relationships method shows off some of the capability of the new API. From a single indicator you can retrieve any related IoCs (in this case we’re using the same file hash as above). In the example here we’re retrieving any known parent processes for the malware that we are investigating.
This returns a DataFrame with the IDs of any related IoCs, their types, and the relationship type to the original observable.
The source column is the ID of our original observable. The target column contains the known execution parents of this file. The results represent a simple graph with the nodes being the source and targets and edges being the relationship_type between them.
Use the limit parameter to this function to restrict how many related entities are returned.
You can look up details of any of the related parents using the get_object() function – but don’t do that just yet until you’ve read the next section.
Looking up multiple observables – lookup_iocs
You can use lookup_iocs to lookup multiple observables in a single call (e.g. IoCs extracted from a set of process events). Assuming that the observables are in a DataFrame called input_df, call lookup_iocs as follows:
results_df = vt_lookup.lookup_iocs(
observables_df=input_df,
observable_column=”colname_with_ioc”,
observable_type_column=”colname_with_vt_type”
)
You can also submit the DataFrame that we generated earlier – example_relationship_df – directly as an input to lookup_iocs. In this case it will default to using the target and target_type columns for the observable_column and observable_type_column parameters, so these do not need to be specified as parameters.
Executing this brings back some basic details about each target item listed in our example_relationship_df DataFrame.
Expanding the Graph – looking up relationships for multiple IoCs
lookup_iocs_relationships is the equivalent of lookup_ioc_relationships but taking multiple IoC/observables as input.
The output from this is shown using the previously generated DataFrame as input. Like lookup_iocs, this is defaulting to using the target and target_type columns for the observable_column and observable_type_column parameters; so we haven’t needed to include these parameters in the call.
You can use this API with an arbitrary DataFrame as input, specifying the observable_column and observable_type_column parameters. The data in both columns must be in the correct format for submission with valid vt_type strings (see the documentation for further details)
Importing the data to networkx
Networkx is probably the most popular Python graphing library. It only includes basic visualization capabilities but it does support a rich variety of graphing analysis and query functions. Since the data structure of the VT relationships is a graph, you can easily import the data into a networkx graph (and also plot it). Having the ability to manipulate the graph in networkx allows you to apply graphing functions and analysis on the data, such as finding the most central nodes, or those having the most neighbors.
We’ve given a simple example of importing the relationships DataFrame into a networkx graph and plotting a simple view of it with Bokeh.
from bokeh.io import output_notebook, show
from bokeh.plotting import figure, from_networkx
from bokeh.models import HoverTool
graph = nx.from_pandas_edgelist(
example_multiple_relationship_df.reset_index(),
source=”source”,
target=”target”,
edge_attr=”relationship_type”,
)
plot = figure(
title=”Simple graph plot”, x_range=(-1.1, 1.1), y_range=(-1.1, 1.1), tools=”hover”
)
g_plot = from_networkx(graph, nx.spring_layout, scale=2, center=(0, 0))
plot.renderers.append(g_plot)
output_notebook()
show(plot)
While this may look cosmic, it isn’t hugely informative and isn’t interactive in any way. On to better things…
Displaying the VT Graph
To see the data in its full glory use the create_vt_graph and render_vt_graph methods:
graph_id = vt_lookup.create_vt_graph(
relationship_dfs=[example_relationship_df, example_multiple_relationship_df],
name=”My first Jupyter Notebook Graph”,
private=False,
)
vt_lookup.render_vt_graph(
graph_id = graph_id,
width = 900,
height = 600
)
You may need to be patient with these APIs since a complex graph can take a while to build and render. In a few moments you should see an interactive graph render into an IFrame in the notebook as seen in the screen shot below.
Exploring the graph on the VirusTotal site brings more capabilities such as being able to view further details about the graph entities, to search within the graph, and to search for, and add, additional nodes.
We very much appreciate this addition to MSTICPy. It brings a lot more of the power of VirusTotal data and exploration to the world of hunting and investigation in Jupyter notebooks.
Recent Comments