Imagine you are a network engineer responsible for supporting an office with 100 people, all of whom use Office 365. One afternoon, all of a sudden, you start to hear persistent complaints from your colleagues that the internet is slow. They can’t load their emails, Word docs, or Excel spreadsheets.
It’s hard to know where to begin. Is the issue due to a faulty wireless AP? Have you lost your internet connectivity? Is a user hogging all the available bandwidth by streaming Netflix? Is Microsoft experiencing a server outage? Or is it all of the above? How do you start troubleshooting? Running packet captures is one option, but it will take a lot of time, which is the one commodity you do not have, since everyone in the office is affected. Lost productivity equals lost money.
This hypothetical scenario became very real a couple of weeks ago in the Cisco Meraki Chicago office. During the afternoon of July 3, a few staff members started observing that Office 365 experience had noticeably deteriorated.
A quick look at Office 365 on the Meraki Insight dashboard suggested that there was a WAN problem affecting O365 traffic.
Closer inspection revealed that the internet connection was fine (i.e. no loss on WAN1 or WAN2), but starting at around 2:30pm, Office365 traffic on the WAN specifically was significantly degraded (~60-80% loss with >3s latency). So much so that almost every user in the office was affected.
Interestingly enough, only our Chicago office was affected where incidentally all traffic is routed through San Jose. This indicated that the root cause was localized to the network path between the Chicago office and the Office365 server local to San Jose it was trying to reach. Thus, in the short term, one workaround could be for a user to VPN into another Meraki location, so that the user’s Office 365 traffic flowed to a different Microsoft server and bypassed the local WAN problems.
Armed with this information from Meraki Insight, our IT team was able to proactively issue guidance to staff at the Chicago office: “Hi Folks, we suspect there’s a WAN outage near an Office 365 server in the San Jose area that is affecting us due to our network traffic being routed to San Jose. I will send an update once I receive more information regarding the outage, however some people have seen success by using VPN to connect to Outlook.”
Indeed, shortly thereafter, Microsoft posted this on their Twitter account, confirming what Insight had indicated.
Office 365 adoption is skyrocketing around the world. Yet, despite this increased adoption, IT admins do not have an easy way to know how or where to start troubleshooting. Without Meraki Insight an IT admin would need to painstakingly test each potential contributing factor one-by-one. This takes time, and since the network conditions keep changing, it becomes impossible to say with certainty which factor or factors is actually causing the problem. Meraki Insight reveals how much each factor is contributing to the problem whilst keeping network conditions constant.
Our customers can spend hours trying to identify the root cause of events like these. Meraki Insight can answer such questions in a matter of minutes, saving them a huge amount of time and money. Learn more and try out Meraki Insight for free here.
Last week, several members from the Meraki product management and product marketing teams huddled in the webinar room at our SF headquarters to present the Meraki Quarterly. The Quarterly takes place every three months and highlights new product innovations that took place over the last quarter. The intent of the Quarterly is not only to keep customers informed about the latest and greatest updates from Meraki, but also to provide customers with an opportunity to get their questions answered by Meraki product experts.
While we were thrilled to see over one thousand registrants for last week’s webinar, we recognize that not all those who registered were able to attend and that some people would prefer a written summary over watching an hour-long webinar. For these folks, here’s a recap of what we discussed.
1. Meraki MV: Small improvements, big impact
The MV smart camera line took a major step forward in April when we introduced the MV32 — our first fisheye camera with the capability to capture 180° of footage — and Motion Recap 2.0, which helps IT admins see motion at a glance by capturing motion in a single image. In the last few months, we’ve made Motion Recap more useful by making the images it captures available in Motion Alert emails and by providing admins the option to disable Motion Recap for bandwidth-constrained networks.
But Motion Recap isn’t the only thing we’ve been working on in the world of MV. We also introduced export checksums, which helps admins ensure that exported footage hasn’t been tampered with, and we extended the retention of exported video to 12 months. Admins now also have the ability to retain captured video when moving a camera from one network to another — e.g., from one office location to another. Finally, a small but useful improvement in the Meraki dashboard is that users no longer lose the tab they’re on (e.g., “Quality and Retention” or “Analytics”) when paging through different cameras.
2. Systems Manager: Playing games and taking names
Customers in every industry use Meraki Systems Manager (SM), Cisco’s official endpoint management tool, to manage devices of all stripes. But there’s one industry that’s particularly excited about SM: education. To help IT admins in education, teachers, and students get excited about SM, we hosted an escape room game at ISTE 2019, the largest K-12 technology show in the US. SM was a key part of the game, with players using SM to solve various puzzles by performing common tasks, like deploying apps and documents to devices.
This past quarter, we also announced a couple of enhancements to SM on the dashboard side. Building and deploying custom profiles is now a lot more scalable and simpler than before thanks to the ability to automate custom Apple profiles with variables; admins no longer have to manually build these profiles one by one. Additionally, end users who want access to corporate email can now upload their own identity certificates through the Self-Service Portal, so IT admins no longer have to create certificates for all their users. These certificates will appear in the Meraki dashboard, so admins will continue to be aware of all the end users with access to corporate email.
3. Why-Fi 6? We’ll tell you
One of our most exciting product launches in recent memory took place this past quarter as we debuted the newest Meraki wireless access points, the MR45 and MR55, equipped with Wi-Fi 6. The new wireless standard is far from a mere spec bump; Wi-Fi 6 is a meaningful step forward that enables higher throughput, higher density, and greater energy efficiency. With features like Target Wake Time, MU-MIMO, and dual 2.4 GHz and 5 GHz radios, the MR45 and MR55 set the standard for the next generation of wireless.
Of course, talking about Wi-Fi 6 isn’t as fun as seeing it deployed live. To that end, during the Quarterly, we highlighted a few real-life deployments of the MR45 and MR55. One of the first deployments of Meraki Wi-Fi 6 was McLaren, the automotive company, where the new APs proved so popular that different teams were moving the APs around to serve their own high density and high throughput purposes. Wi-Fi 6 also proved a popular draw at the US Open, where over 350 of the latest Meraki APs blanketed the course and allowed players and spectators to share, tweet, post, and communicate to their heart’s content.
4. A switch in time saves nine
As any IT admin knows, switches are a crucial part of any network deployment. In the Quarterly, we started by discussing a few key trends we’ve recently observed that are shaping the world of switching: live video streaming & video-first services, more PoE-capable devices, a steady evolution of always-on, power-hungry IoT devices, and inadequate uplink capacity. To address these needs, we just introduced the MS125 access layer switch, which helps admins future-proof their networks by offering 4x10G SFP+ uplinks.
Here’s how the MS125 compares with the MS120 and MS210:
5. Getting Cloudy
Meraki was, of course, born in the cloud, so this is an area of intense excitement for us. First up, this last quarter, we introduced the Meraki Developer Hub and APIs Marketplace, one-stop shops with everything you need to build or buy solutions on top of the Meraki platform. Second, we announced new partner integrations with PagerDuty, Ansible, and OneLogin to help customers make the most of their Meraki deployments. Third, we highlighted action batches and several new endpoints. Finally, we announced that Meraki will be included in a few brand new DevNet certifications coming in early 2020.
That’s a lot of cloud and API announcements! To get a full sense for the Meraki APIs story, sign up for our next Cloud Services and APIs webinar.
6. Security and SD-WAN
Over the last quarter, the Meraki MX team has been hard at work to make our security and SD-WAN appliances more flexible and easier to manage. One of the ways we’ve done that is by debuting a whole new host of API endpoints so developers can use other applications to configure and manage an MX, whether they want to update the MX Layer 7 firewall rules for an MX network or view and update content filtering settings for group policies.
Something we know lots of our customers will be excited about is the news that HTTPS inspection is now in beta. We haven’t yet announced a final release date, but if you’d like to give this feature a try on your own network, contact your sales engineer, sales rep, or Meraki support!
7. Insight into Insight
With Slack and Office 365 recently suffering server outages, we published a couple of blog posts in the last few weeks about Meraki Insight, our network assurance tool. That doesn’t mean our product team wasn’t making Insight better; over the last quarter, we’ve enhanced Meraki Insight with some great new UI improvements designed to make it easier to use and navigate. First, a new Web App Health Details interface improves the troubleshooting experience and helps admins make correlations quicker:
Second, in the WAN Health section, two new fields are available: % capacity, which shows what percentage of upload and download capacity are being used on a particular uplink, and a notes field, which admins can use to take any notes they want about one or more uplinks.
7. Last, but not least
Aside from product updates, we’ve focused on improving the customer experience in a couple of new areas this past quarter. If you haven’t heard already, we have a new podcast, Meraki Unboxed, to give you an inside look at our company. Additionally, the always-thriving Meraki Community recently announced its first set of All-Stars, ten outstanding contributors to our community forum. Congrats to these winners — keep the conversations flowing!
If you made it all the way down here, a sincere thank you for reading all about the latest developments at Meraki. Make sure to tune in to our next Quarterly in October. We don’t want to spoil anything now, but we promise that we’ll have lots more news to share then!
On Friday afternoon, you might have experienced Slack running (a lot) slower than usual. If so, you were not alone. Many people on the Internet noticed that Slack messages and notifications were slightly delayed.
Slack is half-down, so I still have to work, but only half as hard. That's how this works, right?
As always, Meraki Insight detected this issue in real time. Network engineers who were using MI could easily identify the root cause of the problem (hint: it was not the network).
The first sign of trouble was an alert.
If you click on the alert link for further information, you will see that the response time for Slack exceeded 30 seconds (!!) several times during the day.
Finally, MI makes it really simple to identify who is affected. In this case, it turns out that only a few Slack servers were affected, and one domain was especially slow. As a result, not all clients/users were affected, and even those that experienced slowness likely only saw some features working slowly, while the others worked as expected.
Sure enough, later in the day, Slack came out with more details.
In particular, Slack noted that “During this time, approximately 10-25% of jobs resulted in errors or failure. By 10:00 a.m. PDT, we fully restored message delivery and reduced the error rates to less than 5% as the team continued to work on a full recovery.”
An engineer that was using MI would have easily diagnosed this problem in real time and identified why some users were affected while others were not, before Slack posted details of their outage.
Yesterday, the internet was abuzz with the news that Google Calendar was down for several hours worldwide. With over five million businesses large and small reliant on G Suite, this outage had a major impact on workers’ productivity, since people couldn’t check their meetings on the desktop version of Google Calendar. And, of course, there were tweets. So. Many. Tweets.
Since Meraki uses G Suite internally, once we realized that many of our employees couldn’t access their calendars we immediately got to work to investigate the root cause of the issue. Fortunately, thanks to Meraki Insight, our IT team was able to detect the problem in a matter of seconds and begin troubleshooting before users even noticed there was an issue.
Meraki Insight helped our IT team learn, in real time, that Google Calendar was down due to a Google server issue, not because of a problem with Meraki’s LAN or WAN. Having access to this information helped us save multiple hours of investigative work; without knowing that it was a server issue from the start, our team would have frantically been calling our ISP and trying to detect issues with the LAN.
It all started with an email alert, around the time of the actual outage.
By digging deeper, Meraki Insight helped us learn that several servers on Google’s end were partially affected, while two to three were severely affected.
To top it all off, we needed to know how many users on the corporate network were affected by this outage. The short answer: a LOT!
Though SaaS applications are mostly reliable, yesterday’s incident goes to show that they’re never perfect. With more organizations worldwide reliant on cloud-based apps like G Suite, Office 365, and others, the ability to quickly isolate why issues are occurring when they do materialize is crucial. Meraki Insight helps companies know whether a problem is something uncontrollable, like a server outage, or something that can be addressed locally. This kind of visibility can help organizations save an enormous amount of time and sanity — just ask our IT team.
Last month we announced the latest addition to our portfolio of easy-to-manage IT products, Meraki Insight. This advanced new troubleshooting tool will be of interest to any organization working across a wide area network, whether they’re connecting sites together or accessing resources in a remote data center. With more and more IT services migrating from on-site servers to virtual machines running in private or public clouds, it’s never been more important to be able to quickly and easily troubleshoot the WAN and application server performance.
Meraki Insight combines the deep packet inspection capabilities inherent to our MX platform with a robust, mature cloud-hosted architecture to analyze WAN traffic and server response times. By inspecting both the network and application layers, a picture soon emerges, enabling network administrators to quickly identify potential performance bottlenecks that may be leading to a diminished user experience. With IT support tickets being one of the most critical data points for measuring IT effectiveness, anything that helps to accelerate the closing of those tickets and even prevent them is welcome. A better network experience translates to happy users and customers who are more likely to return.
We’ve been delighted with the feedback we’ve received from early customers of Meraki Insight, and not just for the great user experience in the dashboard either. Real issues with real revenue implications are being resolved more quickly already, thanks to the capabilities unlocked by this new tool.
In one case, a well-known retailer with operations around the world was able to use Meraki Insight to rapidly identify and resolve a point-of-sale issue that was impeding sales during its busiest time of year: the holiday season. Another renowned storage company with over 1,500 locations was able to look at WAN performance data to help pinpoint a latency issue that was impacting business-related VoIP calls between branch sites.
Even if alerting isn’t being used to draw attention to an issue, one may be lurking. One of our Meraki partners was about to deliver a demo using their own company’s network when they stumbled across a custom application with a low performance score. All clues pointed to a network layer issue, and sure enough the app was experiencing packet loss severe enough to affect user experience. After a call to their ISP, a support team was dispatched to fix interference on the line.
Real world issues, impacting real business: these are the challenges Meraki Insight helps to address, thereby helping our customers increase revenue and reduce operational costs. For the first time the outstanding LAN troubleshooting tools that have helped build our success at Meraki are looking out to the WAN and the application servers we rely on every day. To gain better insight into issues that may be impacting your network and end-user experience, just click here to initiate a free trial.