Postmortem on the Simple In/Out Outage of March 6, 2023

On March 6, 2023, at approximately 9:30 am Central Standard Time, Simple In/Out experienced a 5-hour outage. Our website was largely unavailable, and our various apps experienced errors. Microsoft Teams presence integrations were also disconnected.

Yesterday was the second-longest outage in our history, rivaled only by the so-called “Netflix” Christmas Eve outage of 2012.

First, allow me to offer an apology. Unlike other outages, this one was entirely our fault. We know many customers use Simple In/Out for mission-critical tasks. Yesterday we let our customers down. We’re sorry for the disruptions during your Monday.

Below we’ll break down what happened, how we resolved the situation, what you can do, and what we intend to do to ensure this doesn’t happen again.

What happened?

As many may have guessed, this was related to the new Quick Picks update we shipped the night before. The update included a shift in how we keep a user’s status visibly current on both the web board and the Microsoft Teams’ Status tab.

A bug in this code made this operation more expensive on our system than anticipated. Another bug deployed simultaneously caused this expensive code to loop in an unanticipated place. Combined, these two bugs would cause an enormous load on our systems. We design our systems for such things, so Simple In/Out worked great for the first 17 hours.

All this changed when we deployed an unrelated bug fix Monday morning. Any time we deploy changes to Simple In/Out, we do so one server at a time. During deployments, we’re always one server short-handed. A non-issue every day but yesterday due to these bugs stressing our servers. This little one-line-of-code unrelated bug fix became the straw that broke the camel’s back.

What followed was a total system collapse. The web servers were overwhelmed with internal requests and started rejecting requests they could not complete promptly. These error responses led to our load balancers disconnecting servers due to too many errors. Next, our DNS provider realized we were throwing excessive errors at our customers and moved to protect us, which led to what many customers saw: scary DNS error pages.

How did we fix it?

We solved these issues by working with our hosting providers to identify bottlenecks (CPU) and spool up more resources. Adding more servers sounds like an easy solution, but we couldn’t definitively say more resources was the solution and wouldn’t worsen the situation. It took most of our outage time to identify the resources impacted. Once we identified the two software bugs that were the real culprits, everything made sense. Fixing those bugs restored the typical workload to our system.

What can I do?

No action is required unless using Microsoft Teams presence integration. Those users will need to reconnect their integrations here.

What are we doing to stop this from happening again?

We will establish more rigorous requirements for new features that increase our server traffic. We will explore more anomaly detection to catch traffic increases that are extraordinary or outside of our typical ebb and flow. We will also add new requirements when taking a server out of rotation during peak hours.

I hope the above deep dive into our operations provides the context behind this outage. We are firm believers in communication and transparency at Simple In/Out. Our worldwide customers trust us, and we work hard every day to maintain that trust. While we failed our customers yesterday, we’ll do our best to ensure it doesn’t happen again.

New Quick Picks Launch for Simple In/Out's Website and Teams

Quick Picks are a core feature, one present during Simple In/Out’s debut back in 2011. Users can, with a single click, set both their status and comment to add valuable context to their in/out board. Quick Picks also improve consistency in updates for reporting purposes.

Our Quick Picks feature has been changing over the past 16 months. We started this transition with Simple In/Out’s app for Android. Simple In/Out Desktop followed as well. Today, our new status update interface comes to simpleinout.com and the Simple In/Out integration for Microsoft Teams.

These changes begin with a new control for updating your status and typing a comment. The old radio buttons have been replaced with a control showing the in/out symbol. This control is now the default status update control, consistently everywhere you can perform a status update.

Simple In/Out Status Update control

Our new Quick Picks interface separates Favorites from Recents. Previously, Simple In/Out had one list containing your favorites and recent updates. This led to confusion because users could only hide a Quick Pick from the list until they next updated their status. Favorites are now 100% curated by the user and the organization. Recents are now in a separate list and are only curated by Simple In/Out. This is the best of both worlds: a Quick Picks list entirely curated by users and a Quick Picks list automatically populated with relevant options.

Simple In/Out Status Update control

In addition to this new status update interface within our Microsoft Teams integration, we’ve added your current Simple In/Out status to the Status tab. No longer will users need to head back to the Board tab to see their Simple In/Out status.

We have only one more application to update with these new interfaces: Simple In/Out for iOS. We’re hard at work and will ship this change to all iPhone users soon.

Microsoft Connectors Restored in Simple In/Out

The Microsoft Teams Connector for Simple In/Out has now been restored. It was missing for the past month and a half. For a debrief on what happened, read on.

At the turn of the year, Microsoft deleted a bunch of Connectors that they claimed had “low traffic”. Among those deleted were connectors owned by a lot of big companies on the internet such as Bitbucket, Google, Mailchimp, Trello, and Zendesk. At first, no one within Microsoft developer support knew what happened. After weeks investigating they discovered their error. By then, they couldn’t get anyone else at Microsoft interested in undoing the damage they had done.

To fix this, we needed to rebuild our Connector from scratch, which we did. But then to pour salt in the wound, a glitch in the App Source process delayed our fix for another 9 days.

The silver lining is that the new Connector is better than ever. We built all configuration into the Connector itself now, which saves time and avoids an extra trip to simpleinout.com to complete the process.

We appreciate everyone hanging with us as we worked through this bug.

Microsoft Has Broken Teams Connectors

Microsoft Teams has long had a feature they call a Connector. Connectors allow third parties, like Simple In/Out, to send messages into the Teams chat channel of your choice. Using Connectors, our customers can relay messages to Teams when users update their statuses.

Connectors are very convenient for organizations that live and die in Teams chat. Coupled with our industry-leading Automatic Status Updates, Connectors can be a powerful tool to keep users up-to-date.

Since January 4th, 2023, Microsoft has broken connectors in Teams.

We began receiving reports from Simple In/Out users that their Microsoft Teams Connectors were no longer working. Worse yet, administrators couldn’t add a new Connector. It’s now impossible for existing customers to replace the broken Connector, and new Simple In/Out trial users cannot test Connectors at all.

We’ve been working with Microsoft since the first report. We’ve filed bug reports, held video calls, and transmitted feedback. Microsoft acknowledges the issue but have no remedy or fix at this time.

To add insult to injury, Microsoft hasn’t publicly acknowledge they have broken Connectors. With no idea of when Connectors will be working again, we have been prodding Microsoft to inform their own customers. Microsoft has avenues for notification, such as their Office Service Health dashboard or the Microsoft 365 Status Twitter account. Both show Teams issues, but this one remains conveniently absent.

When pressed on Tuesday, January 17th, we were promised Microsoft would make this public “very shortly.” After 24 hours of silence, we pressed Microsoft about their definition of “shortly” (which is clearly different from ours). Microsoft responded their “ETA seems Monday night.

You read that right: Microsoft requires a week to publicize a bug they have been investigating for weeks. How long will a fix take if merely tweeting an advisory takes a week? We’re not optimistic.

As you can tell, I’m personally beyond-frustrated with Microsoft’s responses. We know it’s frustrating to our customers relying on Simple In/Out daily. On Microsoft’s behalf, we apologize that this feature stopped working. If you’d like to notified of Microsoft’s slow, glacial, lumbering progress fixing Connectors: reach out to us. We’ll do what they haven’t: keep you in the loop.

The new Simple In/Out TV

This week we’re proud to announce we’ve shipped a major upgrade to Simple In/Out TV. Simple In/Out TV 4 has an all-new interface and offers the most board customization we’ve ever provided in our TV app.

Simple In/Out TV

More Customization

Simple In/Out TV allows you to choose specific user information to display, a quantum leap over the old “our five options only” approach. You can add phone numbers, details, email address, etc. To enable these features, we also needed to allow more precision for text and profile image sizes, as well as control over column counts. We’ve also added advancing filtering by status and multiple groups. as well as multi-column sorting. You can design the TV board of your dreams with Simple In/Out TV 4.

Table View

Simple In/Out TV delivers a new table view alongside our classic card/grid view. Table views are fantastic at displaying tons of information on the screen, all in a single list that’s easy to scan with your eyes. Like our card/grid view, you control the user information displayed, the number of columns, and the sizes of text and images.

Vertical Display

Simple In/Out TV supports both portrait and landscape presentations. If you physically control your environment, you can mount televisions sideways and use Simple In/Out TV to present user lists similar to airport arrival/departure boards (this is my personal favorite way to see user lists).

Simple In/Out TV vertical

Simple In/Out TV 4 is availabe today for download from either the Google Play Store or the Amazon Fire TV Store. We can’t wait to see them in the wild across the globe.

FrontDesk and TimeClock come to Windows 11

Simple In/Out has many great ways to update statuses. For customers that require kiosks, or common-area devices for users to interact with to perform status updates, we have both FrontDesk and TimeClock. Both run on either iPads or Android tablets and handle particular use cases. FrontDesk allows users to see the board and make quick updates with just a swipe or tap. TimeClock is for more controlled environments and works by scanning barcodes that Simple In/Out provides.

The only rub has been that they run on tablets only. Tablets are inexpensive (many even retail for under $100), making them perfect for customers needing new devices. But customers already posessing a Windows computer with a larger touchscreen were out of luck.

Historically, building FrontDesk/TimeClock for these few customers has been tough to justify. Developing new versions of any apps is a resource-intensive commitment that would take our team away from other priorities.

Today, we’re announcing a way forward. With the new Windows Subsystem for Android available in Windows 11, you can now run the Android versions of FrontDesk or TimeClock on a desktop computer. Both are available today in the Amazon App Store within Windows 11. Running on Windows 11 required some tweaks on our end, but we’re happy with the additional flexability this provides our customers around the globe.

We offer instructions on how to install the Windows Subsystem for Android. If you have a large Windows 11 computer already tasked with your in/out board management, you have an exciting new in/out board option.

Automatic Status Updates and Samsung Android 12 Devices

Last month we began receiving reports from some of our Android users that automatic status updates were coming in long after the user entered/left a Geofence. After investigating, it became clear early on that the only users reporting this problem were on Samsung devices with Android 12.

Geofences originally worked great without any additional configuration on Samsung phones running Android 12. As it turns out, this changed in August when Samsung started putting apps to sleep more aggressively. This stops Simple In/Out from launching and immediately processing an automatic status update, instead being launched at a later time. For the purposes of Geofences, this delay is a Samsung introduced problem.

If you are a Samsung phone user on Android 12 and notice that Simple In/Out seems to stop updating your status automatically or if your automatic status updates become significantly delayed, you are in this category of users. We can only hope that Samsung walks this mistake back in future versions of their software.

In the meantime, you can toggle a setting that will stop Samsung from doing this, specifically called Put unused apps to sleep. We’ve added a writeup with screenshots among our Frequently Asked Questions. We’ve also shipped a new version of Simple In/Out for Android that points this setting out in our Automatic Status Update Troubleshooting screen for only Samsung customers running Android 12.

As always, check out our Troubleshooting screen for tips on making Automatic Status Updates work great on your device, Samsung or otherwise. Watch our video on the subject and always feel free to reach out to us directly for assistance.

Send Microsoft Teams Status Updates to Simple In/Out

Simple In/Out’s integration within Microsoft Teams is a big hit. Users living in Teams appreciate having their in/out board at their fingertips all day long.

Historically, there has always been one caveat. Microsoft’s idea of status within Teams and Simple In/Out’s are very different. After all, Simple In/Out shipped nearly seven years before Microsoft Teams. We’ve allowed users to update their Simple In/Out status from within Teams, and users could update their Microsoft Teams status, but the two never spoke to each other.

Today we’re announcing a Public Beta of our integration with Microsoft 365 that will accept status updates directly from Teams. With a few clicks, status update to your Microsoft Teams status will flow into Simple In/Out. Included are status updates made within Microsoft Teams itself, status updates made automatically within Teams (presence, Outlook calendar, phone system integrations, meetings, etc), and temporary updates.

If you have Microsoft Teams open all day long, Simple In/Out will keep everyone up-to-date. Because status updates within Teams transfer to Simple In/Out, all our features are at your disposal. Send followed user notifications directly to your phone when someone updates their Teams status. Utilize Simple In/Out’s plethora of reports to bring accountability to your Teams organization. View all your users and their current Teams status on large displays via Simple In/Out TV. The sky is the limit.

Microsoft 365 status integration for Teams is available today in our Public Beta program. We hope to make this generally available soon, but for now, join our Public Beta program today for access.

Dark Mode for Simple In/Out iPhone

Finally.

Dark Mode on iPhone

Since iOS 13, the iPhone started giving users the preference between light and dark modes. As an iPhone user living in dark mode, I find switching to a “light” app jarring.

While supporting dark mode has always been on our radar, we’ve had other fish to fry with our iOS app. These other projects were always our priority and never left room for this massive redo. That is, until today.

The changes go much deeper than dark mode. We’ve modernized our user interface across the entire app. No more iOS 7-ish blue title bars. We’ve embraced Apple’s esthetic with big titles, semi-transparency, and modern settings menus. Every screen in the new Simple In/Out iPhone app feels new because they were all touched in this redesign.

Take a peek at this before and after:

Before and After

And we’re not done yet. We have plans to bring our status fly-up changes (already live for our Android users) to iPhone in the coming months. Stay tuned.

Drop us an email to let us know what you think any time.

Status Updates from Anywhere with Simple In/Out Desktop

This morning we shipped a new version of Simple In/Out Desktop. In version 1.2 we added a number of new features to make updating your status quicker from anywhere on your computer.

Simple In/Out in your System Tray or Menubar

The most popular feedback we received was allowing users to update their status quickly from anywhere in the app, a behavior that existed in our old desktop apps. Today we’re exceeding that request by placing Simple In/Out right in your Windows system tray or Mac menubar.

Simple In/Out Desktop Tray/Menu

A single click from anywhere on your computer now displays your current status, allows for quickly launching the app, and allows updating your status using your Quick Picks.

This new feature unlocks a handful of other new features as well. Users can now launch Simple In/Out Desktop on startup, in a hidden state, so it’s ready to go. The system tray/menubar will be available for your immediate use.

If you’re using Desktop Presence, launching the app on startup can also begin automatically updating your Simple In/Out status. Because a lot more users will be running Simple In/Out all the time in the background, we’ve brought our Active Time feature to Simple In/Out Desktop. You’re in control of when Simple In/Out Desktop automatically performs status updates.

Right Click to Save Time

We’ve gone further in this release to save managers time by adding right-click menus to the Simple In/Out board.

Simple In/Out Desktop Tray/Menu

Managers can quickly update another user’s status or scheduled statuses without having to click to view the user’s profile first. This shortcut saves a number of clicks, and gives us a place to add more time-saving features in the future.

Update today to the new Simple In/Out Desktop

A reminder that the old Simple In/Out Desktop apps are “end of life” and will no longer allow users to remain logged in consistently starting June 1st. Now is the best time to head to simpleinout.com/desktop and download the new app for Windows/Mac.