Microsoft announced its purchase of the startup MetricsHub which specializes in software to monitor and scale Azure Wed and Worker role instances.   For more information on the acquisition refer to here.

Not sure why it took MS this deep into the Azure lifecycle to obtain the functionality of auto-scaling. It’s the single biggest question IT Ops folks tend to ask when viewing the “Instances” tab in the portal.  At the SDR last year they said auto-scaling was coming very soon so not sure why they would wait this long to finally obtain that functionality.   Purchasing MetricsHub is an interesting but late acquisition for Microsoft. Better late than never I guess…

In this post I will share my initial experience and observations of the MetricsHub tool. I will then compare it to one I use regularly – AzureWatch from Paraleap. Doing the comparison helps to show points of improvement for MetricsHub.

Evaluation of MetricsHub

The MetricsHub site does not give a lot of technical info. It’s high-level and written for marketing folks and even the blogs do not go into detail on how this works and what does and it appears that the functionality of the tool is solely about adjusting scaling based upon rules.  Like Azure Watch it appears to have the ability to monitor endpoints or queue length. The latter is a key factor is loosely-coupled environments for correct scaling.    It also requires an agent (not a favorite of customers) to get metrics on Azure VMs and Websites.  Just to be clear there is no auto-scaling – just metrics – for VMs and Wed sites.

Without the agent the service is 100% free which is very nice. Using normal Azure diagnostics it’s about 50 cent per month charge.  Not sure how they arrived at this number since it uses YOUR Azure storage evidently for WAD tables/blobs.  By contrast, Azure Watch uses THEIR azure storage and you pay them .01 per hour per service to be monitored.  If you turn up their monitoring levels vs. light monitoring levels YOUR storage costs can grow faster and not sure where the .50 cent figure comes from since it appears Azure will be charging up vs. MetricsHub (the cost to log to WAD tables in YOUR storage).  I was very unclear on that.  The .50 cent/month quoted rate to me is a bit misleading. If you are adding approximately .50 per month storage per month, and do not have a policy to truncate after 30 days, the second month your storage bill would be 1.00, 3rd month 1.50, etc.

I like the fact that is has default recommended rules for setup which can be tweaked later. That is a very nice feature since after you install the tool you then have to ‘guess’, based upon experience, what counters to monitor, what their values should be, etc.  For the average Azure customer this is nice.  They also provide alerts in three forms (pager, email, SMS) and give you a detailed breakdown of your monthly bill costs.

When you click signup it takes you to the Azure store.  The only option I was given was to sign up for the “No-Agent” option which is free, select the correct subscription and it’s done. Very simple.  Under contact setting it uses the default login email you use for your Azure portal which you can change but have to first check the box to be emailed promotions to enable the email field. Make the change, and uncheck the promotions box if you want.  The Upgrade button just gives you the original option of free (no agent) so still not sure how to sign up for the Agent option.  The Connection Info button brings up a few “secret” values but not sure where those are used or what I do with them. Obviously used for security purposes but they are not related to my storage key where MetricsHub will store the diagnostics data.

The Manage button is where all the magic happens.  You download the publish settings file first, then upload it to MetricsHub.   All my VMs and Cloud Services are displayed to choose which to monitor.  The VMs are only available using the Agent and still no info on how to sign up for it – these options are not available for me at this point.  Click “start now” to begin the monitoring.

From there you can click on specific roles to view data. You can use the default display or create a custom table or chart.  The default view contains generic CPU, Memory, Disk Read, Disk Write, NetIn, and NetOut values.  The custom table or chart option allows you to only select a subset of the Perfmon counters you have enabled for your application ahead of time.

MetricsHub has a link to Display Issues.  That worked nice when I clicked it as my test app was over 80% CPU and it gave me a message on this. However I did not receive an alert nor did it show up in bold letters on the main page telling me I have an issue. Unless I clicked on the Issues button I did not know about it.

AzureWatch and MetricsHub Comparison

One of the tools I use to monitor Azure applications is the AzureWatch tool. I like it because it is inexpensive, easy to use, has an incredibly powerful rules engine,  and runs from the Web so no setup is needed on your desktop.  Within AzureWatch you aggregate metrics (such as Perfmon counters). Then you write rules against those metrics and define actions to take when those Boolean rules evaluate to true.  Those actions include scaling up or down the number of compute instances, notification options, etc.

So being an AzureWatch OM user I was curious in how MetricsHub matched up with it. Here ar esome of comparisons and observations I made.

Performance Counters

For MetricsHub I did not see an option to add counters beyond what your code or scripts has already told Diagnostics Monitor to track. To me this is a big negative. When I use AzureWatch it allows me to add counters on the fly….this is HUGE for an ITOps person especially during troubleshooting.  To the average customer doing monitoring the ability to add additional counters dynamically is a big feature that’s missing from MetricsHub.

Custom Rules

AzureWatch also allows me to write custom rules and complex logic against those rules. I did not see the option to do that with MetricsHub.

User Interface

The MetricsHub UI is a richer and more user friendly than AzureWatch. However once someone is used to the AzureWatch UI this difference is minimal.

Coverage

In addition to auto-scaling Web/Worker Roles AzureWatch also supports active monitoring of SQL Azure, SQL Federations, Service Bus, Azure Storage and URL’s (active monitoring means they query these resources every minute rather than read some statistical tables). In contrast, MetricsHub only supports Web/Worker Roles without an agent, and Websites/VM’s with an agent running on these machines

Scaling Engine

This is a significant different.  AzureWatch supports scaling to happen via unlimited number of potentially complex rules that can evaluate any amount of metrics of all kinds.  It can auto-scale based upon any kinds of standard or custom performance counters, combinations their-of, service-bus queue/topic levels, storage queues, etc.  In addition, customers can even define their own custom feeds of metrics that can be imported and used for auto-scaling or alerts.  In contrast, MetricsHub appears to only auto-scale based on CPU speed and Storage queues.  AzureWatch can also execute scaling or alerting actions based on a schedule.

Something I was not able to find in Metrics Hub is how to manage how quickly or slowly the scaling occurs. Also there was no way to allow customers truly save money by scaling down at the end of the clock hour.  AzureWatch supports both of these options. This prevents a sudden short spike from causing new instances to be improperly allocated for a non-existent true false surge in load.

Performance Extensions

AzureWatch also has ability to show performance data on mobile devices and RSS feeds and export performance data via PDF/XLS/Word. I didn’t find any such option for MetricsHub.

And the Winner Is…

From what I observed the choice between AzureWatch and MetricsHub lies in the choice between flash and looks vs. power and flexibility. In general, a company whose whole sole business is to provide scaling and monitoring is going to do a better job than a company (Microsoft) who’s simply trying to fill up an array of tools to its customers and check the checkmark to say that it does have monitoring package.  If I was a company considering picking one of the two products, support for monitoring a full variety of Azure middleware and sophisticated scaling engine capable of measuring any amount of indicators with ability to get fancy with complex Boolean logic and order-based rules – that’s what I’d care about.

Overall nothing blows me away with this MetricsHub and there is a lot of functionality it does not provide that AzureWatch give me.  And the winner is…..AzureWatch!

You can install AzureWatch for a free 14-day trial here.

Advertisements