what was I thinking?

[Fadoua]

Good evening everyone,

I decided to share with you my story from today so that you never assume, always check and recheck.

After a long day and when about to log out, my coworker reached out to me asking to update ScriptRunner User License in a dev environment because we just increased the users license. Well without hesitation I started navigating between the different Atlassian Tools to make sure everything was showing the right number. When done, I clicked on BitBucket to update ScriptRunner License. My mistake was I assumed that the link was pointing to Dev when actually it was still pointing to Prod(because we just had a data refresh from Prod). 

This is an unauthorized change because we did have a change ticket for after work hours. We are subject to heavy auditing.

To summarize my situation, I really saw the email coming with my termination. At least I won't have to see HR in-person 🤪

How did I make this stupid mistake? Not sure. Why did I assume? That was another stupid decision.

I hated myself but had to inform my boss. 

Please share with us a situation that made your blood freeze in your veins. It is always good to learn from people mistakes to avoid yourself a similar situation.

Best,

Fadoua

                                              

[JiBrok]

Hello!
This moment is really very exciting - wait, it was prod... @#$%!!!

In my experience, everyone has been in this situation at least once, I call it experience )
I will share my experience.

The Dev stand is completely identical to the Prod. The only difference is a small difference in the url. 

At the end of the day, the amount of time spent at the computer makes your eyes and brain work very strangely. 

I was working with a large workflow that many people often use. I changed several scripts, the logic in transitions, put the plugin and started doing the following tasks.
After a couple of minutes, the world's best monitoring(users) started writing to support. "The process was broken, everything is not working correctly!"

It was very scary, and it is not clear. how I could allow it?! goodbye my career...

But as time went on, the sweat was already pouring out and something had to be done. It is good that everything remained as it was on Dev stand and after 5 minutes it was possible to restore everything without using server backups.

And how good that we did not have such a strict audit then! This was my first job and there was no dismissal. But we discussed the situation with the management :( 

Since then, Dev has specially hung a bright banner on all pages, and Prode has hidden the ability to install plugins and change scripts during working hours with the help of JavaScript.

for example: 

if(new Date().getHours() > 7 && new Date().getHours() < 22){

AJS.$(document).ready(function(){ AJS.$('.upm-update-all-item, #upm-update-all, #upm-upload').remove() });

}

This is not a panacea, but it really helps. 

I hope we never confuse them again))

[Fadoua]

@JiBrok Thank you for sharing your story! I would like to learn more about the change to scripts and hidden install plugin button. Where are you adding the javascript please? I am really interested to get more information if you don't mind.

Best,

Fadoua

[Fadoua]

@JiBrok what do you do in case of an emergency? Since you have the Script.

[JiBrok]

I was adding code to the system announcement banner. Here is an example of code that you can adapt for yourself. 

https://gist.github.com/JiBrok/653b5615199ed991272ab1c8c8345d7f

We store all the code, scripts, and configurations in the git. We check plugins for backward compatibility of data.

Globally, backups of servers and databases are made every day. And in extreme cases, we are ready to lose data during this time. And critical data that is not included in backups will be restored manually.

If only the functionality is affected, we simply roll it back and check that the data is in the correct state.

If the data is broken, we try to fix it (manually, using scripts, or restore it from backup).

We have a large instance(15k users) but we do not have a strict audit. So problems with functionality are solved by simply reinstalling the previous version.

In banks, I have seen systems around Jira that allow you to make any changes to the prod only through CI/CD systems. You first configure everything on the test environment, and then roll them out on the prod. This applies to any changes - fields, scripts, wf, projects, plugins... It looks cool and complicated. For ourselves, we have so far abandoned such a system - it is good for auditing and controlling functions on the prod, but it does not insure against errors.

To be honest, I've never heard of anyone being fired because of mistakes. Mistakes make us better. But maybe I was just lucky and I didn't meet such companies :)

[Fadoua]

@JiBrok Greatly appreciated, thank you very much!