Friday fun: Biggest screw-up at work edition Edited

Everyone at some stage of career, would have experienced screwing up something at work. Sometimes it go unnoticed and sometimes it makes our day worst. However, down the lane when we recollects the incident and the way we overcame it, we find ourselves a laughing stock. :P I could remember a few regarding JIRA itself.

Once I mis-mapped Issue Key in my CSV to Issue Type in JIRA while importing. JIRA created 1500+ new issue types on import and it went down. Hahaa.. However, my then lead @Vishnukumar Vasudevan [ACP-JA] was kind enough and was cool, though he knew it from the support team via logs. Luckily it was a development instance and it was brought up again. :P

There was another incident where I was writing some groovy scripts. I was referring to the JIRA java APIs navigating from google page (without noticing the version of Java doc :P). During production go-live (on an upgraded version), it started throwing lot of errors on logs since a few methods were deprecated. Somehow we fixed it sitting overnight. :D

Note: Please don't share these with your current team members/managers ;) :P

We do not learn anything new by doing everything right! Make mistakes, learn from them, move on!! Happy Screwing-up!! :)

37 comments

Oh oh, time to confess ;-)

My biggest screw-up has nothing to do with Atlassian's applications. Once, I wanted to install and test a new Oracle OCI Driver to be used by our applications. Of course, locally, on my computer, nowhere else...

But since I am an administrator with wide-reaching authorizations, the installation cleared and updated our central installation. Without warning, nothing. The installation found the environment variable TNS_ADMIN on my machine, the path pointed to our central installation and things started...

I wasn't expecting anything bad, but soon after the installation succeeded, I heard upset discussions on the hallway, more and more and more people couldn't connect to the database anymore. Our staff couldn't work. And then I realized, that my installation was not as local as I thought it would be. It removed our central database configuration and installed a new standard one.

We had to restore the configuration from yesterday's backup and after another twenty stressful minutes, everything was fixed...

Like • 2 people like this

Oh god.. I could feel the heat! Anyway cheers!! :)

OMG!!! That way a serious screw-up!!

My heart just sank for you while I read this.

Like • 1 person likes this

In my first job after college I was deleting a record from a user account SQL table ... I missed highlighting the where clause. So, what got executed was "delete from users". I indeed deleted the record I wanted to delete ... and every other record too. After about five minutes of freaking out and finally fessing up to my manager we restored from backup and I learned a valuable lesson. I now check every delete statement about 50 times before running it and I do it in a transaction so that I can rollback if the number of records deleted was not what I expected.

Like • 2 people like this

Yup. Done that. The missing where clause is an SQL pit trap

hahaa

Like • 1 person likes this

Oh missing the where clause is a sin!

Reminds me of my screw-up and learning for life.

I have done that once using a script and screwed a complete table. Luckily it was a dev table and no users were disturbed. However, my face went pale and I couldn't think of anything for some time. The best part, being a dev instance it did not have backup and I really did not what to tell my fellow developers coz it had some 1M records that we had collected during a course of 3-4 days for a BI work and now we had a blank table. My team was very understanding and I was saved!

However, from then on, I look at the query at least a fifty times before executing it. And when I script them my precautions are much more.

What a learning!!!

I used to take backup of the tables whenever I do alter commands and major updates to avoid such disasters. :D It was a master advice from a peer who has done such mistakes in his past.

Missing where clauses it's not a pit trap, it's just lack of common logic. I work mostly with MS SQL SERVER and before launching every modifying command (create, alter, update, delete) I prepare all the statements I need, double check them and prepare some other statements to check if the outcome is good. Then enclose then in a beautiful

"BEGIN TRANSACTION" and I put also a ROLLBACK and after a COMMIT TRANSACTION.

If everything is good I proceed with the commit, If i was trying to delete a single record and I see "102210 rows affected" after launching it, I promptly launch the ROLLBACK command and feel immediately better.

@Mede Made A better practice! thanks :)

I’ve done this too...good job I had the same table data but on a replica site!

Love this topic, @Mahesh S! I hope you don't mind that I updated your headline to include the "Friday fun" phrase. 😄

oh okay Thanks :)

CSV imports sure can make a big mess. That's why I always want to do them in staging first.

yeah that's a better move! :)

@Mahesh S I did not know you screwed up with the issue type this big. It would have been fun right!!!

@Arun Thundyill Saseendran Yes very much funny, but not at that moment! :D

There is the classic unexpected space at the command line. Instead of typing

rm -rf /foo/bar/baz

to delete just the baz directory, you type

rm -rf / foo/bar/baz

with a space after the first slash. And destroy the whole machine! This is why I type

ls / foo/bar/baz

first to check for errors, then up arrow and edit the line to use something other than ls.

Posted on behalf of a friend, of course :)

yeah.. there's something to learn here! :D

I remember reading a story of a medium scale Web Data Center admin who for automation learned ansible and automated cleaning some of the files. He did the same mistake of having a space after / in the path and literally wiped off the complete data center and his business.

My biggest mistake was also my smallest!

In the days we did DevOps without calling it that, I looked after a small trading system and its test copies. A script I'd made a mistake in ran on schedule and over-wrote production data with an older version.

Biggest mistake - it wiped £40 Billion (yes, billion) off the market value of the company, causing a worrying blip in all the stock markets, fortunately corrected manually as soon as it was spotted

Smallest mistake - it was one "s" missing in the script. If I'd got the wrong character or added an extra one, the script would have failed and done nothing.

Lessons:

- never neglect your unit tests

- always be honest about the mistake 1 - the Chairman was able to correct the blip by issuing a statement saying "our trading system fell over, use the last known value until we get it fixed".

- always be honest about the mistake 2 - it would have taken days to find if I had not been honest about it (there was no way it could have been traced to me), but we were able to fix it in minutes once we'd traced it. We did spend the rest of the day making 100% sure we'd got back to the right data everywhere. (Shame VMS died, it would have taken a week to recover from the inferior Windows and *nix filesystems we all rely on nowadays)

Wow. (Love the lessons learned!)

Don't think I would have survived the heart attack to go on and correct the issue. You're stronger than I!

I would have lost it @Nic Brough [Adaptavist], I will start packing my stuff before the boss hit the door.

hahaaa.. @Meg Holbrook

I didn't have time to have a heart attack - I was stuck on a train heading into the office when I got the call. I was able to diagnose it within seconds as well!

OMG @Nic Brough [Adaptavist]! Seriously you are a strong man and appreciate your courage to accept the mistake. I believe it in - "Any mistake is forgivable if one has the courage to accept it"

Great learnings!

Wow lots of great stories shared, love this Friday fun topic! Mine was during Security week when I completed a quiz for prizes, and I “replied all” to my department with all my answers to the quiz. Grateful for the retract email feature but some had fast fingers and opened it within a second. Now I’m so careful to never hit the reply all button 😆

I know this is a low risk example but gratefully I can’t think of any other screwups, at least none that I’m aware of to date. 😊

lol.. you nearly made the quiz cancelled or ending up changing the questions!! However congrats for the prizes. ;)

LOL @Mahesh S - yes I was mortified! Luckily it was just for fun, but still can't believe I did it!

Mine was slightly different. I sent to a huge mailing list a 'half email' with all bits and pieces. I had the habit of drafting the email in the mail client and it was very embarrassing.

From then on, I started following two things

1. I draft my emails in notepad and have placeholders for graphs or images.

2. When I hit a reply all, I type some junk like 'adasdadad' into the to list in outlook so that even if I hit a send, outlook will not send saying it is an invalid email id. Only after I re-read I remove the junk and hit send.

I have a mailing story too -on the Apple mail system (unlike Windows outlook), sending out of office auto-replies is a tricky business. You have to deliberately search for and select a checkbox for something like "send only for future emails". I ended up spamming everyone with multiple emails -one auto-reply for any email the poor people had sent to me in the past! I was ready to die of shame! Well, at least I provided entertainment to folks and got them all together in cursing the flood of emails :)

I told this story in my admin class at Summit and won 'worst story of the room', it's still my most shameful moment:

I was a baby admin, a few months into our onboarding of JSD, JIRA, and Confluence. We had all the basics down and had been running smoothly. We hadn't correlated the difference between done and resolved, and at some point elected to clean up our 13,000+ tickets to reflect the correct status.

I should also mention that I had NOT followed best practices for notifications (they were all still set to defaults, holy moly, so many notifications).

I batched every single one of those 13000+ with a status change and then reports from all over the company started rolling that emails were not sending/receiving. It didn't take long for me to realize that I'd crashed our entire company's email server (after thousands of emails had already spammed every user who'd ever sent a ticket in).

I owned up immediately and we took mitigation steps asap. Still embarrassing, still the moment that gives me pause whenever I'm doing batch updates, ugh.

I nearly did this too with less impact during a bulk update. :D

@Mahesh S I remember it!!! I can still feel the heat...!!

I love this topic!

I made a change in a Field Configuration Scheme within 5 min, the Sr. Atlassian Admin started getting emails, calls and even in-person visits from people wondering what was going on!!! How come a field that was never required before is now blocking them from creating their tickets?

You know who got the nasty look!!!!

From that day I was asked to make changes in Dev server only and stay away from JIRA PRD until I feel strong enough to work in it.

Oh no! I remember having that exact same issue but with email requests into JSD because it didn't occur to me that, duh, someone that was emailing wouldn't be able to fill that field.

Doesn't it feel awful when the realization first dawns on you?

After reading our stories, it will be hard for others to believe that we are champions now. :D

I found a group picture:

LOL

@Mahesh Swe are at least dealing with Apps and Servers, think about the ones who make a mistake and end up somebody's life.

hahaa.. lol!

@Mahesh S that was blunt... :-P

But as your opening statement reads, we cannot learn without mistakes.

Making mistakes is OK. But not learning from them and repeating them is a SIN.

@Arun Thundyill Saseendran .. Very much true! Here, I could see most of the screw ups from folks whom I thought as best in the field. Make mistakes and happy learning!

I don't have anything as big to tell.

During my apprenticeship I was supposed to tidy up an workshop on the IT Service Department's floor. It needed tidying up because a table collapsed (under the weight of to many PCs and notebooks; That's another story).... When the table collapsed it unplugged many PCs and other devices from power and ethernet sockets.

So in my goodwill I plugged them all back in. I didn't really care what cables belonged to what devices, I just thought since this is the Room for maintenance and setup I plugin everything back in including network.

And with this I hecked up the whole companies network. I plugged in a tiny access point / router device. It was meant for an external internet subscription, separate from the companies network; to test Notebooks for "homeoffice" usage. Just simulating a simple home wifi... But I didn't know.

This small box took over the whole Network and to handle DHCP; Rendering a few connections and applications useless, killing all the printer using DHCP and so on.
This for the whole branch of this company (around 2000 people).

I accidentally launched a Rogue DHCP attack with a consumer grade router. It took networking one week to find the router. And longer to find out I was the culprit. ¯\_(ツ)_/¯

After that I knew that network administration is definitely not the direction I'm taking in my career.

I am laughing so hard. We just learned about Rogue DHCPs in my network security class! May I share this story?

Sure go ahead. :)

:-D

When you add a project role to a notification scheme for the issue created events, and a newbie project admin then adds jira-users to the project role. Which means that everyone in the company gets email when an issue is created in that project.

This spammed 15,000 users including the CEO of a well-known company a few years ago.

We ended up creating a default group with no-one in it, named the group "Do not add large groups here" and made it a default project role member. But really I'd like Jira to check the size of groups before using them to send email to.

I hacked the .vm file that shows the groups in the sharing screen, so it never presented the jira-users and other groups people had been spamming with!

Nice. Now I think it's all javascript calls pulling the groups and users from the server.

My mother's favorite thing to say is "There's always two versions of a story; the long and the short version. The only difference is adjectives."

Short Version: I caused by best friend's dad to end up in a car wash with no car because I didn't didn't speak Spanish and my Sister didn't speak Chinese.

Long Version:

So we had this stupid system at the car wash I worked at. My best friend's parents owned it. They're both from China and speak Mandarin at home. My sister at the time also worked there, and she speaks Mandarin fluently. I took Spanish in high school for a year. The line workers (guys in the tunnel) were pretty much all from central america and spoke Spanish. Everyone at least spoke extremely broken English. The stupid system being, there were two buildings, the car wash and the auto shop that were basically on two sides of a gas station. My sister worked above the gas station in accounting, where there was a phone. There wasn't one in the auto shop. If one of the line workers had a problem, they'd inform me, and I'd call over to my sister in accounting and tell her, then she'd basically lean out her window and shout at Kevin (boss / best friend's dad) who was usually in the auto shop.

Well while my sister was getting her minor in Chinese, Kevin decided he was going to make her only talk to him in Mandarin. So one day, while we were closing, one of the older guys who spoke pretty much ZERO English I think told me that I couldn't go get the fregona because the soap was stuck. It took me a few moments to realize what I thought he said: I needed to go get the mop from where we kept the car wash soap.

In my defense, it made sense. I wasn't allowed in the car wash tunnel, and we stored the mop in the car wash. It was usually by the dryers in the tunnel. I'd never been in there, I assumed there were big vats of soap or something. So I told my sister. She attempted to tell Kevin. Kevin, knowing I wasn't allowed in the tunnel, went to go get the mop for me, grumbling on how the line workers should've brought it to me. So he went into the tunnel... and about thirty seconds later the car wash turned on and he came out screaming drowning in old soap and wax.

Apparently the guys were purging the soap out since something had clogged... and that someone would bring the mop by when they were done... at least according to the one guy who spoke English that for some reason they didn't decide to send.

I'm still not allowed in the car wash tunnel and I still don't know where the soap is kept.

Like • 2 people like this

The short version is so great I don't even want to spoil it by reading the long version!

BUT I WILL.

Like • 2 people like this

Here's a community management screwup story. I managed the online community at a giant tech company whose name rhymes with Schmoracle. We had test spaces that often had to be deleted after the testing was complete. I filed a dev ticket and pasted in the URL of the test space to be deleted.

Thanks to a cut and paste error, I actually pasted a URL for the corresponding live space. The dev who picked up the ticket should have caught it (because why would I be deleting a live space full of community content) but he didn't. So then, click click, the live space was deleted.

Restoring from backup would have meant rolling back the entire past 24 hours of community posts. Since our community was so large, both internally and externally, I would have had to explain to many many many other stakeholder groups why my cut-and-paste error meant they would lose 24 hours of content. At least one of these stakeholder groups was led by an abusive executive and that conversation would have been... unpleasant.

I came up with a Hail Mary solution to restore the backup to our test server and then manually rebuild the deleted community via cutting and pasting from test to production and then reformatting all their content when all the formatting was screwed up. It took the team two weeks of manual work to fix my mistake. (I of course did nothing else for two weeks except help fix it.)

The team whose community space was deleted was understanding about it (!) and appreciative that I had found a solution (!!) and said "wow I hope nobody gets in trouble for this" (!!!).

I immediately implemented a new system where we would "archive" areas before deleting them as a double-check, so it would never happen again. But I probably gained dozens of gray hairs in the process.

That's an amazing screw up story, but even more amazing is how you managed to pull a solution out of your pocket and save the day.

The best feeling is when you break bad news to a client and they trust you enough to let you work out a solution.

Achievement Get: Superhero

Oh I feel your stress from here! I'm so so happy that team was understanding. That makes issues like this sooo much better to admit and fix your mistakes.

rofl! It would be funny to explain people what caused this! But, as Heth said, you were lucky for your team! :)

LOL!!! That's an amazing screw-up!!! 24 hours of content!!! Disastrous for a large community... Cool that you came up with a solution and your team backed you and help you restore.

My biggest mistake was an accident. When I was very small, my little sister came to visit us and had a sleepover. While my sister is showering, I went into the bathroom and stepped on a soap. I tripped over my mom's cosmetic shelf, almost all my mom's cosmetic was damaged. Then, the soap fly into my sister's opening mouth and she swallowed it. It sounds pretty dramatic, but it is absolutely true.

After that, my sister was at the hospital. And my mom was very angry that the cosmetic she brought was destroyed...........

You almost committed a murder! LoL!! :D

Wow... I still remember that incident @Mahesh S :)

lol! :P

A few years back, I got my first coding job as an intern. The job entailed working on a rather large scale health project, that supported about 5,000 hospitals. A few weeks in and having proven my "prowess" in code, I was given credentials to the live server db (was meant to copy live data for testing), and was told to remove some data duplicates. The duplicates were over 10000 entries in the DB, and I could not manually remove them. So I quickly did a script to get duplicates and tested locally. Worked like a charm. Having 'tested', I ran the script on the live DB and left for home at eleven am, with the days work done.

I was called back half an hour later because of the reports having disparities and that's when I knew i had messed up. Worth pointing out that the tool was being used for reporting and the reports were used as a KPI for the users. A few were almost fired that day.

The error was that in the deletion script, I forgot to skip the first entry, and that my local copy did not contain the duplicates as it were dummy data. Thankfully we had a backup and we were able to resolve the issue without any bloodshed

Haha.. Another nightmare on the db query!! :)

Well in the YouTube age of videos showing people trying to drive a 10 Ton JCB onto a small wooden ferry and being surprised when both sank, we probably need to rethink the scope of screw up!

In 2nd place for me was during a live systems migration that had to run old and new in parallel from pre-month-end until month end, after which the new would take care of itself.

At end of each business day I received that day’s transactions in my transformation environment and my nights work would begin.

I’d examine the data and generate script after lousy script to gradually transform it piece by lousy piece. Around 5am I’d be done, detach the database, and move it to the network along with the test evidence, ready for ingress into the new system later when ops came in, about 8am.

They had to take the new data, run new systems previous day end for the imported data, then start of new day so It was ready for use by 9am. Then I’d head home and try to get some sleep.

After 3 or 4 days of this I’m nearly a zombie. I finished as usual around 5am. My head is thinking detach database, but my fingers hear DROP database. Oops.

I had 3 hours to do it all again. Coat off, coffee on.

Fortunately, part of the testing protocol, required by the auditors, bless em, was to paste the SQL script and a results summary for each of the transformation processes into the testing evidence document, so I managed to get it finished just the very second the ops walked in.

In 1st place, and a much shorter story, turns out it’s not a good idea to date the girl who also happens to be the mistress of the guy who owns the company. Oops, a literal screw up.

Like • 1 person likes this

My "favorite" screw-up happened in the nineties, on a Friday afternoon.

I wrote a perl daemon that monitored an application. It sent out a status e-mail to an SMS gateway, that routed the message to my mobile (a Nokia 2110 at that time, which could hold about 5 texts or so...)

Unfortunately, I have managed to put the sending in an infinite loop.

It took me about half a minute to realize before I had killed the script. However, it had already sent out a *lot* of e-mails by then... all queued at the (external) gateway.

There was no way to undo. I kept receiving and deleting texts for three days straight.

lol!

Well, another one :)

Early in my career, I have learnt two things:

- If you point a domain to an Apache web server, and you don't set up a virtual host for it, the default host will still server it.

- When you install a web server, and you create the first index.html to test it, it is perfectly enough to write "It works" into it. You do not have to postfix that with "a**holes!".

This thread is triggering distant memories! This one didn’t exactly happen to me, but I had a walk-on part in it. It was back in the mid 80’s on IBM midrange systems and goes under the category “when all the stars line up right, or wrong”

A junior programmer had been given the task of adoing user error messages for a new system that was under development and almost complete and this was one of just a few remaining tasks. It was boring, mechanical and repetitive and rather than just plough through it he was trying to automate it, but couldn’t get his idea working and the IT manager was at him about deadlines and getting it ready for demo.

I was on a different project, but In my coffee break I went to chat with him to find out what the problem was. He showed me what he was trying to do. I knew a way to do it much easier and just had time to show him. I explain how the system actually used message files, showed him how to create his own message file, add an error message to it and then override the system file for his application. He looked on in wonder as his message appeared in response to some bad input from the screen. All he had to do was add all the required error codes and messages and he would be done. I left him to and went back to my own work.

Late in the afternoon, there was a sudden commotion across the hall, raised voices, then angry shouting, then slamming doors, then silence. I went to their project room and it was empty.

I finished up my work, it was almost 5 o’clock and headed down to our social club on the beach which also served as the extended meeting project room, and most of the team were there, all talking about Roger.

Roger owned the company but didn’t spend much time there anymore. I’d only seen him once. His surname was foreign, and translated to English as rooster or cock, so inevitably he was known by the nickname Roger the cock, but only at the social club, never at work.

They explained the junior programmer had been sent home and suspended, and John, the IT director (we had simple titles back then), was going to decide what to with him once the compulsion to kill him had worn off. What did he do? I asked.

Turns out, he had experimented a bit with adding the new messages and figuring out how to re edit them, and during this learning phase got bored with the dull error messages and added his own. He preceded each error with “Roger’s cock up:” followed by some innuendo related to the actual error such as “ don’t put it like that, it hurts”, you get the idea.

Impressed by his wit, he tried to see how many other of the messages he could make witty, the answer, it seems, was quite a lot and that’s how he spent the rest of his morning, thinking about all the rumors and stories he’d heard about the infamous Roger in the social club and seeing if he could match them with an appropriate, or rather, inappropriate message.

In the afternoon the manager was on his case again and he explained the issue was resolved, he was now adding in the messages, he was nearly done and it would be finished by end of business. Presumably he meant the real messages would be done by then.

The manger just caught the IT director, John, on his way up to the Board Meeting that was happening that afternoon, and ever eager to impress, informed him that the demo system would be ready today.

As it happened for this particular Board meeting, Roger had decided to make one of his rare appearances. Eager to impress, particularly with Roger there, John used his turn to inform the board that the new system was ready for Demo.

As the meeting drew to a close Roger said that as he was in the building today, could he see a demo of the new system. John was rightly proud of the system, it was a good team and they’d built a nice system, he was delighted Roger wanted to see it. And of course, if Roger was interested, the other directors were interested too.

So the board meeting reconvenes back down in the project room and John starts to demo. I’m trying to imagine the look on the poor programmers face at this point.

And as John gives the demo, I’m guessing he made some input errors, and I think that must be when the shouting started.

That's so hilarious ...

🤣

This might not have been the most earth-shattering error, but one I shall never forget.

While attending a military course one of my mathematical answers came to .67.

My end of the exam mark was 99.

When I inquired, "Why? Where did I go wrong?"

I was told that I should have written 0.67. The "0" is an integral part of such a solution.

More than 60 years have passed since then and the episode is still fresh in my mind.

My mark for the following week's exam was 100 - after learning that "0" = "1".

Learning from experience :P

What a lesson! Even 0 is important.

at least you didn't a `where` clause in an update query

Ok. Here I go. We had a internal monitoring application which is used widely to see real time stats. I managed the full stack of that application - it had MySQL cluster database. On a Thursday afternoon, I was implementing a new feature and testing that out in a test environment setup in my local machine. Instead of dropping the tables from test database, I dropped them on production.

Imagine 50+ emails of "can't monitor anything" in 15 mins. I panicked and blood drained out of my face. To my horror, the backup script wasn't working - the last backup was a month old. I called my manager and explained him and he was like "Man - how did you do that ?". He told me not to worry and see if anything could be done.

Logs. Logs saved me that day. I had the application write lots of logs - from database queries to HTTP requests and all in between. It dawned on me after a strong coffee. I wrote a script to parse all the logs and re-construct the schema and data. Around 11 PM, I had re-constructed around 80% of data, which brought back the application in a "usable" state.

Next day, I implemented a double backup strategy, and revoked the permission to delete / alter / drop anything from production. From that day, last ten mins of the day is dedicated to checking daily and hourly backups.

Now I understand why my PO asks me to check whether the backup script is working on a daily basis. I believe he would have faced something similar!! :-)

Log save us!

Giff's to this thread is a lot funnier than the actual content.

Few of the above mistakes had been done by many of my team members and you know after doing that they came to me for a fix as I am junk who know many things, the mistakes made and I could fix it:

1- executed rm with space after slash--- No fix server has gone completely.

2- Opened the malware which encrypts the hard drive on test server -- No fix had to format it.

3- Deleted the entire transactions from account table in production forgot to check where condition. -- Restored from latest backup though lost few records.

4- Executed chmod on root directory lost the os.

these are the big mistakes done by my colleagues and team members and many other minor :)

Easy.

1) I wrote an application that was not 1998-compliant.

2) I was a developer on a global call centre sales system. There were always problems with restocking inventory if a transaction was cancelled, and I was assigned to fixing it. I sorted it easily, it sailed past testing and everybody was very pleased.

We went out to Paris to install for the sales team, and all was going well. 9am, the first rep takes a call and has the inventory screen up. However, none of the other 30 reps can open the screen. I had put a transaction round the order screen which locked the DB and prevented any other reads. I'd turned a 30 user sales order system into a Single User one.

And I didn't have the source code with me.

Was like the first twenty minutes of Saving Private Ryan.

Was like the first twenty minutes of Saving Private Ryan.

omg this analogy.... hahahaha

OMG :)

@John Duncan [Administrator] What happened after that? How did you fix it?

After furiously wondering for an hour whether we could do something at a DB level (and deciding we couldn't), we instigated our tried and tested rollback procedures: reinstall the previous version (We had the Installshield-built setup disk) and manually fixed the DB schema changes.

We also hired a second tester so we could do multi-user...

Thanks for sharing! I needed this today - this was my kind of day today.

Comment

Off-topic

Products

Interests

Groups

Community resources

Support

Friday fun: Biggest screw-up at work edition Edited

37 comments

Comment

Jonny Adams

Friday fun: Happy Pi-day!

Atlassian User Groups

Find my local user group

Groups near you

Ask a question

Start a discussion

Products

Interests

Groups

Community resources

Support

Friday fun: Biggest screw-up at work edition Edited

37 comments

Comment

Jonny Adams

Friday fun: Happy Pi-day!

Atlassian User Groups

Find my local user group

Groups near you