Friday fun: Biggest screw-up at work edition Edited

[Mahesh S]

Everyone at some stage of career, would have experienced screwing up something at work. Sometimes it go unnoticed and sometimes it makes our day worst. However, down the lane when we recollects the incident and the way we overcame it, we find ourselves a laughing stock. :P I could remember a few regarding JIRA itself.

Once I mis-mapped Issue Key in my CSV to Issue Type in JIRA while importing. JIRA created 1500+ new issue types on import and it went down. Hahaa.. However, my then lead @Vishnukumar Vasudevan [ACP-JA] was kind enough and was cool, though he knew it from the support team via logs. Luckily it was a development instance and it was brought up again. :P 

There was another incident where I was writing some groovy scripts. I was referring to the JIRA java APIs navigating from google page (without noticing the version of Java doc :P). During production go-live (on an upgraded version), it started throwing lot of errors on logs since a few methods were deprecated. Somehow we fixed it sitting overnight. :D

Note: Please don't share these with your current team members/managers ;) :P 

We do not learn anything new by doing everything right! Make mistakes, learn from them, move on!! Happy Screwing-up!! :)

[Thomas Schlegel]

Oh oh, time to confess ;-)

My biggest screw-up has nothing to do with Atlassian's applications. Once, I wanted to install and test a new Oracle OCI Driver to be used by our applications. Of course, locally, on my computer, nowhere else...

But since I am an administrator with wide-reaching authorizations, the installation cleared and updated our central installation. Without warning, nothing. The installation found the environment variable TNS_ADMIN on my machine, the path pointed to our central installation and things started...

I wasn't expecting anything bad, but soon after the installation succeeded, I heard upset discussions on the hallway, more and more and more people couldn't connect to the database  anymore. Our staff couldn't work. And then I realized, that my installation was not as local as I thought it would be. It removed our central database configuration and installed a new standard one.

We had to restore the configuration from yesterday's backup and after another twenty stressful minutes, everything was fixed...     

[Mahesh S]

Oh god.. I could feel the heat! Anyway cheers!! :)

[Davin Studer]

In my first job after college I was deleting a record from a user account SQL table ... I missed highlighting the where clause. So, what got executed was "delete from users". I indeed deleted the record I wanted to delete ... and every other record too. After about five minutes of freaking out and finally fessing up to my manager we restored from backup and I learned a valuable lesson. I now check every delete statement about 50 times before running it and I do it in a transaction so that I can rollback if the number of records deleted was not what I expected.

[Matt Doar (LinkedIn)]

Yup. Done that. The missing where clause is an SQL pit trap

[Mahesh S]

hahaa

[Erica Moss]

Love this topic, @Mahesh S! I hope you don't mind that I updated your headline to include the "Friday fun" phrase. 😄

[Mahesh S]

oh okay Thanks :)

[Matt Doar (LinkedIn)]

CSV imports sure can make a big mess. That's why I always want to do them in staging first.

[Mahesh S]

yeah that's a better move! :)

[Matt Doar (LinkedIn)]

There is the classic unexpected space at the command line. Instead of typing

rm -rf /foo/bar/baz

to delete just the baz directory, you type

rm -rf / foo/bar/baz

with a space after the first slash. And destroy the whole machine! This is why I type 

ls / foo/bar/baz

first to check for errors, then up arrow and edit the line to use something other than ls.

[Matt Doar (LinkedIn)]

Posted on behalf of a friend, of course :)

[Mahesh S]

yeah.. there's something to learn here! :D

[Nic Brough [Adaptavist]]

My biggest mistake was also my smallest!

In the days we did DevOps without calling it that, I looked after a small trading system and its test copies.  A script I'd made a mistake in ran on schedule and over-wrote production data with an older version.

Biggest mistake - it wiped £40 Billion (yes, billion) off the market value of the company, causing a worrying blip in all the stock markets, fortunately corrected manually as soon as it was spotted

Smallest mistake - it was one "s" missing in the script.  If I'd got the wrong character or added an extra one, the script would have failed and done nothing.

Lessons:

- never neglect your unit tests

- always be honest about the mistake 1 - the Chairman was able to correct the blip by issuing a statement saying "our trading system fell over, use the last known value until we get it fixed".

- always be honest about the mistake 2 - it would have taken days to find if I had not been honest about it (there was no way it could have been traced to me), but we were able to fix it in minutes once we'd traced it.  We did spend the rest of the day making 100% sure we'd got back to the right data everywhere.  (Shame VMS died, it would have taken a week to recover from the inferior Windows and *nix filesystems we all rely on nowadays)

[Erica Moss]

Wow. (Love the lessons learned!)

[Meg Holbrook]

 Don't think I would have survived the heart attack to go on and correct the issue. You're stronger than I!

[Fadoua M. Boualem]

 I would have lost it @Nic Brough [Adaptavist], I will start packing my stuff before the boss hit the door.

[Mahesh S]

 

hahaaa.. @Meg Holbrook

[Nic Brough [Adaptavist]]

I didn't have time to have a heart attack - I was stuck on a train heading into the office when I got the call.  I was able to diagnose it within seconds as well!

[Jodi LeBlanc]

Wow lots of great stories shared, love this Friday fun topic! Mine was during Security week when I completed a quiz for prizes, and I “replied all” to my department with all my answers to the quiz. Grateful for the retract email feature but some had fast fingers and opened it within a second. Now I’m so careful to never hit the reply all button 😆

I know this is a low risk example but gratefully I can’t think of any other screwups, at least none that I’m aware of to date. 😊

[Mahesh S]

lol.. you nearly made the quiz cancelled or ending up changing the questions!! However congrats for the prizes. ;)

[Jodi LeBlanc]

LOL @Mahesh S - yes I was mortified! Luckily it was just for fun, but still can't believe I did it!

[Meg Holbrook]

I told this story in my admin class at Summit and won 'worst story of the room', it's still my most shameful moment:

I was a baby admin, a few months into our onboarding of JSD, JIRA, and Confluence. We had all the basics down and had been running smoothly. We hadn't correlated the difference between done and resolved, and at some point elected to clean up our 13,000+ tickets to reflect the correct status. 

I should also mention that I had NOT followed best practices for notifications (they were all still set to defaults, holy moly, so many notifications).

I batched every single one of those 13000+ with a status change and then reports from all over the company started rolling that emails were not sending/receiving. It didn't take long for me to realize that I'd crashed our entire company's email server (after thousands of emails had already spammed every user who'd ever sent a ticket in). 

I owned up immediately and we took mitigation steps asap. Still embarrassing, still the moment that gives me pause whenever I'm doing batch updates, ugh. 

[Mahesh S]

I nearly did this too with less impact during a bulk update. :D

[Fadoua M. Boualem]

I love this topic!

I made a change in a Field Configuration Scheme within 5 min, the Sr. Atlassian Admin started getting emails, calls and even in-person visits from people wondering what was going on!!! How come a field that was never required before is now blocking them from creating their tickets?

You know who got the nasty look!!!!

From that day I was asked to make changes in Dev server only and stay away from JIRA PRD until I feel strong enough to work in it.

[Meg Holbrook]

Oh no! I remember having that exact same issue but with email requests into JSD because it didn't occur to me that, duh, someone that was emailing wouldn't be able to fill that field. 

Doesn't it feel awful when the realization first dawns on you? 

[Mahesh S]

After reading our stories, it will be hard for others to believe that we are champions now. :D

[Meg Holbrook]

I found a group picture: 

[Fadoua M. Boualem]

LOL

[Fadoua M. Boualem]

@Mahesh Swe are at least dealing with Apps and Servers, think about the ones who make a mistake and end up somebody's life.

[Mahesh S]

hahaa.. lol!

[Niklas Becker (bitvoodoo ag)]

I don't have anything as big to tell. 

During my apprenticeship I was supposed to tidy up an workshop on the IT Service Department's floor. It needed tidying up because a table collapsed (under the weight of to many PCs and notebooks; That's another story).... When the table collapsed it unplugged many PCs and other devices from power and ethernet sockets.

So in my goodwill I plugged them all back in. I didn't really care what cables belonged to what devices, I just thought since this is the Room for maintenance and setup I plugin everything back in including network.

And with this I hecked up the whole companies network. I plugged in a tiny access point / router device. It was meant for an external internet subscription, separate from the companies network; to test Notebooks for "homeoffice" usage. Just simulating a simple home wifi... But I didn't know.

This small box took over the whole Network and to handle DHCP; Rendering a few connections and applications useless, killing all the printer using DHCP and so on.
This for the whole branch of this company (around 2000 people).

I accidentally launched a Rogue DHCP attack with a consumer grade router. It took networking one week to find the router. And longer to find out I was the culprit. ¯\_(ツ)_/¯ 

After that I knew that network administration is definitely not the direction I'm taking in my career.

[Heth Siemer]

I am laughing so hard. We just learned about Rogue DHCPs in my network security class! May I share this story?

[Niklas Becker (bitvoodoo ag)]

Sure go ahead. :)

[Matt Doar (LinkedIn)]

When you add a project role to a notification scheme for the issue created events, and a newbie project admin then adds jira-users to the project role. Which means that everyone in the company gets email when an issue is created in that project. 

This spammed 15,000 users including the CEO of a well-known company a few years ago.

We ended up creating a default group with no-one in it, named the group "Do not add large groups here" and made it a default project role member. But really I'd like Jira to check the size of groups before using them to send email to.

[Nic Brough [Adaptavist]]

I hacked the .vm file that shows the groups in the sharing screen, so it never presented the jira-users and other groups people had been spamming with!

[Matt Doar (LinkedIn)]

Nice. Now I think it's all javascript calls pulling the groups and users from the server.

[Heth Siemer]

My mother's favorite thing to say is "There's always two versions of a story; the long and the short version. The only difference is adjectives."

Short Version: I caused by best friend's dad to end up in a car wash with no car because I didn't didn't speak Spanish and my Sister didn't speak Chinese. 

Long Version:

So we had this stupid system at the car wash I worked at. My best friend's parents owned it. They're both from China and speak Mandarin at home. My sister at the time also worked there, and she speaks Mandarin fluently. I took Spanish in high school for a year. The line workers (guys in the tunnel) were pretty much all from central america and spoke Spanish. Everyone at least spoke extremely broken English. The stupid system being, there were two buildings, the car wash and the auto shop that were basically on two sides of a gas station. My sister worked above the gas station in accounting, where there was a phone. There wasn't one in the auto shop. If one of the line workers had a problem, they'd inform me, and I'd call over to my sister in accounting and tell her, then she'd basically lean out her window and shout at Kevin (boss / best friend's dad) who was usually in the auto shop.

Well while my sister was getting her minor in Chinese, Kevin decided he was going to make her only talk to him in Mandarin. So one day, while we were closing, one of the older guys who spoke pretty much ZERO English I think told me that I couldn't go get the fregona because the soap was stuck. It took me a few moments to realize what I thought he said: I needed to go get the mop from where we kept the car wash soap.

In my defense, it made sense. I wasn't allowed in the car wash tunnel, and we stored the mop in the car wash. It was usually by the dryers in the tunnel. I'd never been in there, I assumed there were big vats of soap or something. So I told my sister. She attempted to tell Kevin. Kevin, knowing I wasn't allowed in the tunnel, went to go get the mop for me, grumbling on how the line workers should've brought it to me. So he went into the tunnel... and about thirty seconds later the car wash turned on and he came out screaming drowning in old soap and wax.

Apparently the guys were purging the soap out since something had clogged... and that someone would bring the mop by when they were done... at least according to the one guy who spoke English that for some reason they didn't decide to send.

I'm still not allowed in the car wash tunnel and I still don't know where the soap is kept.

[Monique van den Berg]

The short version is so great I don't even want to spoil it by reading the long version!

BUT I WILL.

[Monique van den Berg]

Here's a community management screwup story. I managed the online community at a giant tech company whose name rhymes with Schmoracle. We had test spaces that often had to be deleted after the testing was complete. I filed a dev ticket and pasted in the URL of the test space to be deleted.

Thanks to a cut and paste error, I actually pasted a URL for the corresponding live space.  The dev who picked up the ticket should have caught it (because why would I be deleting a live space full of community content) but he didn't.  So then, click click, the live space was deleted.

Restoring from backup would have meant rolling back the entire past 24 hours of community posts. Since our community was so large, both internally and externally, I would have had to explain to many many many other stakeholder groups why my cut-and-paste error meant they would lose 24 hours of content. At least one of these stakeholder groups was led by an abusive executive and that conversation would have been... unpleasant.

I came up with a Hail Mary solution to restore the backup to our test server and then manually rebuild the deleted community via cutting and pasting from test to production and then reformatting all their content when all the formatting was screwed up. It took the team two weeks of manual work to fix my mistake. (I of course did nothing else for two weeks except help fix it.) 

The team whose community space was deleted was understanding about it (!) and appreciative that I had found a solution (!!) and said "wow I hope nobody gets in trouble for this" (!!!). 

I immediately implemented a new system where we would "archive" areas before deleting them as a double-check, so it would never happen again. But I probably gained dozens of gray hairs in the process. 

[Meg Holbrook]

That's an amazing screw up story, but even more amazing is how you managed to pull a solution out of your pocket and save the day. 

The best feeling is when you break bad news to a client and they trust you enough to let you work out a solution. 

Achievement Get: Superhero

[Heth Siemer]

Oh I feel your stress from here! I'm so so happy that team was understanding. That makes issues like this sooo much better to admit and fix your mistakes.

[Mahesh S]

rofl! It would be funny to explain people what caused this! But, as Heth said, you were lucky for your team! :)