It's not the same without you
Join the community to find out what other Atlassian users are discussing, debating and creating.
Everyone at some stage of career, would have experienced screwing up something at work. Sometimes it go unnoticed and sometimes it makes our day worst. However, down the lane when we recollects the incident and the way we overcame it, we find ourselves a laughing stock. :P I could remember a few regarding JIRA itself.
Once I mis-mapped Issue Key in my CSV to Issue Type in JIRA while importing. JIRA created 1500+ new issue types on import and it went down. Hahaa.. However, my then lead @Vishnukumar Vasudevan [ACP-JA] was kind enough and was cool, though he knew it from the support team via logs. Luckily it was a development instance and it was brought up again. :P
There was another incident where I was writing some groovy scripts. I was referring to the JIRA java APIs navigating from google page (without noticing the version of Java doc :P). During production go-live (on an upgraded version), it started throwing lot of errors on logs since a few methods were deprecated. Somehow we fixed it sitting overnight. :D
Note: Please don't share these with your current team members/managers ;) :P
We do not learn anything new by doing everything right! Make mistakes, learn from them, move on!! Happy Screwing-up!! :)
In my first job after college I was deleting a record from a user account SQL table ... I missed highlighting the where clause. So, what got executed was "delete from users". I indeed deleted the record I wanted to delete ... and every other record too. After about five minutes of freaking out and finally fessing up to my manager we restored from backup and I learned a valuable lesson. I now check every delete statement about 50 times before running it and I do it in a transaction so that I can rollback if the number of records deleted was not what I expected.
Love this topic, @Mahesh S! I hope you don't mind that I updated your headline to include the "Friday fun" phrase. 😄
There is the classic unexpected space at the command line. Instead of typing
rm -rf /foo/bar/baz
to delete just the baz directory, you type
rm -rf / foo/bar/baz
with a space after the first slash. And destroy the whole machine! This is why I type
ls / foo/bar/baz
first to check for errors, then up arrow and edit the line to use something other than ls.
My biggest mistake was also my smallest!
In the days we did DevOps without calling it that, I looked after a small trading system and its test copies. A script I'd made a mistake in ran on schedule and over-wrote production data with an older version.
Biggest mistake - it wiped £40 Billion (yes, billion) off the market value of the company, causing a worrying blip in all the stock markets, fortunately corrected manually as soon as it was spotted
Smallest mistake - it was one "s" missing in the script. If I'd got the wrong character or added an extra one, the script would have failed and done nothing.
Lessons:
- never neglect your unit tests
- always be honest about the mistake 1 - the Chairman was able to correct the blip by issuing a statement saying "our trading system fell over, use the last known value until we get it fixed".
- always be honest about the mistake 2 - it would have taken days to find if I had not been honest about it (there was no way it could have been traced to me), but we were able to fix it in minutes once we'd traced it. We did spend the rest of the day making 100% sure we'd got back to the right data everywhere. (Shame VMS died, it would have taken a week to recover from the inferior Windows and *nix filesystems we all rely on nowadays)
I would have lost it @Nic Brough [Adaptavist], I will start packing my stuff before the boss hit the door.
Wow lots of great stories shared, love this Friday fun topic! Mine was during Security week when I completed a quiz for prizes, and I “replied all” to my department with all my answers to the quiz. Grateful for the retract email feature but some had fast fingers and opened it within a second. Now I’m so careful to never hit the reply all button 😆
I know this is a low risk example but gratefully I can’t think of any other screwups, at least none that I’m aware of to date. 😊
I told this story in my admin class at Summit and won 'worst story of the room', it's still my most shameful moment:
I was a baby admin, a few months into our onboarding of JSD, JIRA, and Confluence. We had all the basics down and had been running smoothly. We hadn't correlated the difference between done and resolved, and at some point elected to clean up our 13,000+ tickets to reflect the correct status.
I should also mention that I had NOT followed best practices for notifications (they were all still set to defaults, holy moly, so many notifications).
I batched every single one of those 13000+ with a status change and then reports from all over the company started rolling that emails were not sending/receiving. It didn't take long for me to realize that I'd crashed our entire company's email server (after thousands of emails had already spammed every user who'd ever sent a ticket in).
I owned up immediately and we took mitigation steps asap. Still embarrassing, still the moment that gives me pause whenever I'm doing batch updates, ugh.
I love this topic!
I made a change in a Field Configuration Scheme within 5 min, the Sr. Atlassian Admin started getting emails, calls and even in-person visits from people wondering what was going on!!! How come a field that was never required before is now blocking them from creating their tickets?
You know who got the nasty look!!!!
From that day I was asked to make changes in Dev server only and stay away from JIRA PRD until I feel strong enough to work in it.
@Mahesh Swe are at least dealing with Apps and Servers, think about the ones who make a mistake and end up somebody's life.
I don't have anything as big to tell.
During my apprenticeship I was supposed to tidy up an workshop on the IT Service Department's floor. It needed tidying up because a table collapsed (under the weight of to many PCs and notebooks; That's another story).... When the table collapsed it unplugged many PCs and other devices from power and ethernet sockets.
So in my goodwill I plugged them all back in. I didn't really care what cables belonged to what devices, I just thought since this is the Room for maintenance and setup I plugin everything back in including network.
And with this I hecked up the whole companies network. I plugged in a tiny access point / router device. It was meant for an external internet subscription, separate from the companies network; to test Notebooks for "homeoffice" usage. Just simulating a simple home wifi... But I didn't know.
This small box took over the whole Network and to handle DHCP; Rendering a few connections and applications useless, killing all the printer using DHCP and so on.
This for the whole branch of this company (around 2000 people).
I accidentally launched a Rogue DHCP attack with a consumer grade router. It took networking one week to find the router. And longer to find out I was the culprit. ¯\_(ツ)_/¯
After that I knew that network administration is definitely not the direction I'm taking in my career.
When you add a project role to a notification scheme for the issue created events, and a newbie project admin then adds jira-users to the project role. Which means that everyone in the company gets email when an issue is created in that project.
This spammed 15,000 users including the CEO of a well-known company a few years ago.
We ended up creating a default group with no-one in it, named the group "Do not add large groups here" and made it a default project role member. But really I'd like Jira to check the size of groups before using them to send email to.
My mother's favorite thing to say is "There's always two versions of a story; the long and the short version. The only difference is adjectives."
Short Version: I caused by best friend's dad to end up in a car wash with no car because I didn't didn't speak Spanish and my Sister didn't speak Chinese.
Long Version:
So we had this stupid system at the car wash I worked at. My best friend's parents owned it. They're both from China and speak Mandarin at home. My sister at the time also worked there, and she speaks Mandarin fluently. I took Spanish in high school for a year. The line workers (guys in the tunnel) were pretty much all from central america and spoke Spanish. Everyone at least spoke extremely broken English. The stupid system being, there were two buildings, the car wash and the auto shop that were basically on two sides of a gas station. My sister worked above the gas station in accounting, where there was a phone. There wasn't one in the auto shop. If one of the line workers had a problem, they'd inform me, and I'd call over to my sister in accounting and tell her, then she'd basically lean out her window and shout at Kevin (boss / best friend's dad) who was usually in the auto shop.
Well while my sister was getting her minor in Chinese, Kevin decided he was going to make her only talk to him in Mandarin. So one day, while we were closing, one of the older guys who spoke pretty much ZERO English I think told me that I couldn't go get the fregona because the soap was stuck. It took me a few moments to realize what I thought he said: I needed to go get the mop from where we kept the car wash soap.
In my defense, it made sense. I wasn't allowed in the car wash tunnel, and we stored the mop in the car wash. It was usually by the dryers in the tunnel. I'd never been in there, I assumed there were big vats of soap or something. So I told my sister. She attempted to tell Kevin. Kevin, knowing I wasn't allowed in the tunnel, went to go get the mop for me, grumbling on how the line workers should've brought it to me. So he went into the tunnel... and about thirty seconds later the car wash turned on and he came out screaming drowning in old soap and wax.
Apparently the guys were purging the soap out since something had clogged... and that someone would bring the mop by when they were done... at least according to the one guy who spoke English that for some reason they didn't decide to send.
I'm still not allowed in the car wash tunnel and I still don't know where the soap is kept.
Here's a community management screwup story. I managed the online community at a giant tech company whose name rhymes with Schmoracle. We had test spaces that often had to be deleted after the testing was complete. I filed a dev ticket and pasted in the URL of the test space to be deleted.
Thanks to a cut and paste error, I actually pasted a URL for the corresponding live space. The dev who picked up the ticket should have caught it (because why would I be deleting a live space full of community content) but he didn't. So then, click click, the live space was deleted.
Restoring from backup would have meant rolling back the entire past 24 hours of community posts. Since our community was so large, both internally and externally, I would have had to explain to many many many other stakeholder groups why my cut-and-paste error meant they would lose 24 hours of content. At least one of these stakeholder groups was led by an abusive executive and that conversation would have been... unpleasant.
I came up with a Hail Mary solution to restore the backup to our test server and then manually rebuild the deleted community via cutting and pasting from test to production and then reformatting all their content when all the formatting was screwed up. It took the team two weeks of manual work to fix my mistake. (I of course did nothing else for two weeks except help fix it.)
The team whose community space was deleted was understanding about it (!) and appreciative that I had found a solution (!!) and said "wow I hope nobody gets in trouble for this" (!!!).
I immediately implemented a new system where we would "archive" areas before deleting them as a double-check, so it would never happen again. But I probably gained dozens of gray hairs in the process.
This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.
Read moreConnect with like-minded Atlassian users at free events near you!
Find a groupConnect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.
Start an AUGYou're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs