Tentative title: A sniper aimed down the receiver of a McDonalds drive thru microphone.
Back when I worked at Telstra (or for boomers who seem to more commonly refer to it as Telecommunications Australia) as a consumer phone and internet plan salesman, a co-worker of mine once walked into our back office and informed our store manager "Hey, do you remember how you mentioned we all get our one big fuck up? I think I just had mine and am going to need help".
The "fuck up" in this instance was quoting a customer a free home router valued at $216 they weren't entitled to due to being a recontract for a home internet deal rather than a brand-new connection. Far from what I think constitutes the term as I'm sure these things were made for $20 a pop.
But anyway, it's a statement that's kind of stuck by me no matter the job, and trust me, there have been more than one big fuckup at some of these. For context I have three university degrees in technology but all of them were achieved before the real advent of cloud computing and SaaS services, nowadays I'm pretty sure the content of them would be useless. Honestly a lot of my learning has been through trial, error and faking it till you make it (short for: read every piece of documentation and try to not let it overwhelm you).
What story am I to tell?
- The time I applied a terrible fix to a set of remote desktop servers that ended up killing 200 workstations?
- Working at an MSP where I was in charge of an on-prem to cloud migration with a freak oversight which caused massive burnout?
- When I called out fraud practices at a workplace and copped massive heat for it?
- Or many more...
I think I'll start with the MSP one.
For the uninitiated, an MSP is otherwise technically known as a "Managed Services Provider", or as I like to unashamedly explain to everyone "You hire us when your company is too cheap to buy their own IT staff". They've become a more popular alternative to internal IT through the years, and I feel that everyone has had a run in with them at some point. And it's appealing, isn't it? You pay 80 grand for a systems administrator, or you pay 50 grand a year and you get access to like 10 IT guys of varying knowledge, granted none of them are on site.
So here I am. It's early to mid 2020, and the pandemic has just begun. I'm working at an MSP who has taken on way, way more clients than I think they can chew. One of these clients is a transport and logistics company, about 20 staff. One thing to keep in mind with the pandemic is that it quickly forced companies to adopt remote working procedures. This meant laptops, VPNs, Home setups, Et cetera. And for some companies in particular, typical of blue-collar workspaces, who maintained on premise infrastructure and services were not at all prepared for the culture shock about to be set upon them.
So here I am, three months into my job and I'm told that I have to migrate 20 employees at this logistics company from an on-premise exchange to a 365 based environment. Easy, right?
Caveat being that I can't actually reformat any of the office installations on these devices. So, here's the plan: we spend a few minutes with each employee, and we ask them what their workflow is like and what they use. I assign a level two tech to go and investigate half while I go do the other half.
It all seems swell. We're prepared to go. Using BitTitan we queue up a prelim migration of all their mail and services, once done I remote in to set up AzureAD joining to their local domain controller (keep in mind, they had no secondary DC). Upon setting it up, I realize a problem, the idiot in me forgot that passwords from 365 propagate downwards and overwrite that of the local accounts.
No big deal it just means I have to call their contact on site and give them their new passwords, and a guide to reset them (while sending the level 2 tech to go take care of remoting in and guiding through the technically inept how to do it). Easily done. A 5-minute phone call, then I tell my boss what's up and all is well, mission accomplished, bill em.
Except that's not how it goes.
About a good 30 minutes after that phone call and reassuring my boss that all was done and to hit them with the bill, I started getting a phone call from the owner of this logistics company. Upon answering, I hear an absolutely pissed off old man, shouting at the top of his lungs, swearing down the microphone of his phone and exclaiming that nothing is working. What? Nothing changed in the last 30 mins and I got the all clear that things were fine before. What the fuck.
I call our contact on site and ask what's going on, he explains that Employee A doesn't have access to Employee Bs mailbox. What? We didn't scope any shared mailboxes, and of those that we did see, permissions shouldn't have been affected. WRONG!
You see, in a classic case of "My nephew is good with computers", this old man got his nephew to set up outlook for everyone ages ago, and this MSP was needed as this young kid had gone to uni - or some shit. What this kid had set up, was not shared access to mailboxes. He had gone in and manually signed into other people's accounts on each outlook instance with their usernames and passwords, this old man had the user and pass of everyone in the company. So of course, a password change for one person, would remove access for everyone.
Cue a phone call from the old man again, my level 2 tech had given him MY PERSONAL MOBILE NUMBER, and he was now shouting down the McDonalds drive thru quality microphone right into my ears. It felt like a sniper rifle was aimed at the receiver of his phone and was ready to send a 50 caliber lead injection through copper and satellite signals, direct into my brain at any moment. Bullseye.
Hanging up was no luck, my service delivery manager had told me that should I push back or duck any calls, I was as good as gone.
This was so fucked. On top of that, disappointment coming down from my service delivery manager and boss told me enough that we weren't billing extra for this. Or rather, that I wasn't getting any extra billable hours.
What do you do in this situation? I did the responsible thing... at first.
I called and remoted into desktops, set up shared mailboxes and the like. Do it right. But then heard from Mr Dickhead Owner (name slightly changed for anonymity) that many employees were off today. Out of the twenty that we had migrated, only four were on roster that day. No way into those remaining desktops.
New instruction from my boss: "Just fix it"
Going against the mantra of "Do it once, do it right, never do it again"; I reset the passwords of all remaining 16 employees again, And I set it up so that they wouldn't have to reset the password on next login. I then called our rep at the client, and informed him of all these passwords, I sent it to him in a list, and explained that. If anyone got a prompt for a user's password, the corresponding password was on the list send via email to our rep.
The angry calls vanished, and everything was quiet.
Two days passed.
A post incident review of this event took place in the company meeting room. The boss, the delivery manager, the level two tech, and I. Expecting to get a major hour long reeming, I prepared for the worst. But instead got told two things.
- In spite of what had happened, we had gotten a nine out of 10 review.
- "Good fix in the heat of the moment. Well done!"
What?
"Good fix?". I later learned there was no desire for us to go back and correct this glaring issue with mailbox access permissions as it could quote unquote off-the-record "lead to more billable hours from them, they're an ad-hoc customer after all".
What did I learn from this experience?
Honestly. I still maintain this isn't something that could have been properly scoped. The direction was terrible, and the fix was terrible too. It also highlighted some shitty MSP business practices (the lack of desire to actually improve upon existing systems).
Take this as a lesson in what not to do if you want to be a respectable IT rep.
I left the MSP world two months later. How could I stay somewhere knowing of the multiple bombs ready to randomly go off at the height of a worldwide disaster in the making (the pandemic).
... Only to make the mistake of joining another MSP seven months later. But those, are stories for other times.
This story is dedicated to D, who laughed for ages when I told her the drive-thru mic sniper punchline. Miss you ❤️
