Confluence server giving 504 gateway error due to "valves.StuckThreaddetectionValve.notifyStuck"

[Sanjay Chinthala]

Hi Team,

We are getting the below error and then confluence server become unresponsive 

and please provide the any solution to overcome the below issue.

26-Mar-2019 07:35:34.033 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8090-exec-1323] (id=[175625]) was previously reported to be stuck but has completed. It was active for approximately [1,423,015] milliseconds. There is/are still [9] thread(s) that are monitored by this Valve and may be stuck.
26-Mar-2019 07:35:34.033 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8090-exec-1264] (id=[171233]) was previously reported to be stuck but has completed. It was active for approximately [159,827] milliseconds. There is/are still [8] thread(s) that are monitored by this Valve and may be stuck.
26-Mar-2019 07:35:34.033 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8090-exec-1335] (id=[175637]) was previously reported to be stuck but has completed. It was active for approximately [1,395,154] milliseconds. There is/are still [7] thread(s) that are monitored by this Valve and may be stuck.
26-Mar-2019 07:35:34.033 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8090-exec-1294] (id=[173914]) was previously reported to be stuck but has completed. It was active for approximately [277,131] milliseconds. There is/are still [6] thread(s) that are monitored by this Valve and may be stuck.
26-Mar-2019 07:35:34.033 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8090-exec-1322] (id=[175624]) was previously reported to be stuck but has completed. It was active for approximately [1,105,237] milliseconds. There is/are still [5] thread(s) that are monitored by this Valve and may be stuck.
26-Mar-2019 07:35:34.033 WARNING 

Thanks,

Sanjay

[Shannon S]

Sanjay,

Thank you for reaching out to us about this issue. I'm happy to help you look into this and see what the problem might be.

From what you've pasted, I don't actually see any error messages, but what looks like a series of warnings. From your Confluence server logs you'll be able to find more details including error messages (will start with the word ERROR instead of WARNING) as well as what is causing the error.

It will also help to know what stats your server is running on, including:

• Database server and version
• Confluence version
• Java memory settings (See How to Fix out of Memory Errors for more details on where to find that)
• When this last worked normally, and if you made any changes since then.

Thank you for your help!

Regards,

Shannon

[Sanjay Chinthala]

Hi Shannon,

Please find following details

1.Database server and version: postgresql 9.6

2. Confluence version:   confluence 6.13

3. server configuration

total memory: 4gb

java memory allocated: 1GB(CATALINA_OPTS="-Xms1024m -Xmx1024m -XX:+UseG1GC ${CATALINA_OPTS}")

4. Confluence working normally and we are getting this issue every 30 days once

from the logs, not able to find any errors

Thanks,

Sanjay

[Shannon S]

Hello Sanjay,

It might be the low memory being allocated, but I'd like to have a look at what comes up in the log scanner. You can find this in the Support Tools & Plugins from your instance.

If you have trouble with that, feel free to send along your log file. You can do so using Dropbox here, and I can scan it for you.

Regards,

Shannon

[Sanjay Chinthala]

Hi Shannon,

we have memory 4gb on the server and currently, java heap size allocated to confluence is 1gb and can we increase to 2gb

Thanks,

Sanjay

[Shannon S]

Hello Sanjay,

Thank you for providing those details.

4GB memory total is not sufficient for Confluence, as we recommend a minimum of 6GB:

• Server Hardware Requirements Guide

Minimum hardware requirements

The values below refer to the minimum available hardware required to run Confluence only; for example, the minimum heap size to allocate to Confluence is 1 GB and 1 GB for Synchrony (which is required for collaborative editing). You'll need additional physical hardware, of at least the minimum amount required by your Operating System and any other applications that run on the server.

 On small instances, server load is primarily driven by peak visitors, so minimum system requirements are difficult to judge. We provide these figures as a guide to the absolute minimum required to run Confluence, and your configuration will likely require better hardware.

Here is our minimum hardware recommendation:

• CPU: Quad core 2GHz+ CPU

• RAM: 6GB

• Minimum database space: 10GB

You'll need to change to a server with at least 6GB-8GB RAM, and then you can try setting the java heap to 2GB. 

Please let me know how that goes!

Shannon

[Sanjay Chinthala]

sure, we will do that

[Shannon S]

Great, thank you for confirming, Sanjay!

I'll await your response. Let me know if you need any help migrating!

Regards,

Shannon

[Sanjay Chinthala]

@Shannon S We are planning to migrate confluence server to confluence cloud

can you please provide the steps for migrating from confluence server to confluence cloud 

Are we need to Setup SSL after migrating to confluence cloud for the domain, is it maintained by confluence cloud?

after migrating confluence data to confluence cloud is data will delete automatically on the original server?

[Shannon S]

Hello Sanjay,

You will want to review the document Plan your Confluence Server to Cloud migration prior to migration.

We do have a tool called Cloud Migration Assistant for Confluence that can help you with your migration.

If you run into any issues at all during the migration, please raise a case directly with our Cloud Support Team.

You will not need to set up an SSL after migrating, everything is fully maintained by Atlassian. The only thing you want to be sure of is to create your own backups regularly. 

Once you've migrated to Cloud and can confirm that the site is working as expected, let your users know to start accessing the site via the Cloud URL. You can then decommission your Server instance. You'll need to do this manually on your end. 

I hope that helps! Let us know if you have any questions about it.

Regards,

Shannon

[Sanjay Chinthala]

Hi Shannon,

While running the migration getting the below errors and can you please help me on this

46% - An unexpected error occurred during step: Migrating users and groups. Error: Failed to add user to groups

Thanks,

Sanjay

[Sanjay Chinthala]

running the new plan fixed the issue :)

[Shannon S]

Hi Sanjay,

Thank you for letting me know. Can you just confirm what you mean when you say you ran the new plan? I just want to make sure it's clear for community members if they run into the same issue.

Regards,

Shannon

[Sanjay Chinthala]

Hi Shannon,

when I run migration it failed and after running the migration again by creating the new plan it is working fine 

Initial error:

An unexpected error occurred during step: Migrating users and groups. Error: Failed to add user to groups

1. After completion of total migration if we run the migration again will it migrate total data again?

2. can we point our confluence server domain to atlassian confluence cloud url?

3. for transferring data from confluence server to confluence cloud have any cost?

4. any new change in space in confluence server will it migrate to confluence cloud while running the second plan?

Thanks,

Sanjay

[Shannon S]

Hello Sanjay,

Thank you for confirming that.

To answer your questions:

1. Creating a new plan will not overwrite data. If you need to re-run the migration on any data you already migrated, then you will need to delete the migrated data and then do it again.
2. To point your server domain to Confluence Cloud, you will need to set up a redirect. You can do this with an HTML redirect by embedding it in your header (Confluence Administration > General Configuration > Custom HTML, under Look and Feel). You can also use an add-on, such as Redirection for Confluence. 
3. For transferring the data, the Cloud Migration Assistant for Confluence is free. There is no cost for you to complete the migration with that add-on. 
4. If you make a change in your space on the Confluence server, you will need to delete it from the migrated Confluence Cloud site and create a new plan. Any recent changes on the Cloud instance which are not on your Server instance will be lost.

I hope this helps! Let me know if you have any questions about this.

Regards,

Shannon

[Sanjay Chinthala]

Hi Shannon,

I am not able to choose the new cloud site for migration

when I clicked on "choose cloud site" and it gives the below error and I am using chrome browser

error:

We tried to load scripts but something went wrong.

Please make sure that your network settings allow you to download scripts from the following domain:

https://aid-frontend.prod.atl-paas.net/atlassian-id/front-end/4.5.0

Thanks,

Sanjay

[Shannon S]

Sanjay,

Thank you for providing that screenshot. Have you given that URL to your network administrators? They can make sure that your network isn't blocking access to that domain.

If they can confirm it isn't being blocked, then I'd ask for you to clear your browser cache and then try to login to http://id.atlassian.com/. Let me know if you are able to!

Regards,

Shannon

[Sanjay Chinthala]

Hi Shannon,

This is happened due to the firewall issue and I have admin access for 2 confluence cloud site while I am planning to migrate to second confluence cloud site it not showing in the list

Thanks,

Sanjay

[Shannon S]

Hello Sanjay,

I had a look at the email address that you are you using here, but I don't see that you're Site Admin for either of the Cloud sites that come up. You need to be Site Admin or you won't be able to use the tool with that site.

If you still have issues with this, I would really encourage you to raise a ticket with the Cloud Support Team and they can help you with this.

Regards,

Shannon

[Sanjay Chinthala]

.

[Shannon S]

Hello Sanjay,

I let you know on Wednesday how you can do that:

To point your server domain to Confluence Cloud, you will need to set up a redirect. You can do this with an HTML redirect by embedding it in your header (Confluence Administration > General Configuration > Custom HTML, under Look and Feel). You can also use an add-on, such as Redirection for Confluence. 

Note that you cannot do a DNS redirect, it must be a redirect directly to the URL of your Cloud instance.

Regards,

Shannon

[Sanjay Chinthala]

.

[Shannon S]

Sanjay,

You are redirecting your Server instance (confluencetest.dimagi.com) to your Cloud instance (confluencetest100.atlassian.net).

As I suggested before, this needs to be done either with an HTML Redirect, or you use Redirection for Confluence, an add-on. If you choose the add-on, then you need to keep Confluence running on the Server instance.

If you choose to do the HTML redirect, it can be done within Confluence, by embedding it in your header (Confluence Administration > General Configuration > Custom HTML, under Look and Feel).

You can also do it outside of Confluence if you no longer wish to keep Confluence running on the server. If you choose that route, then you will want to create an index.html file at your domain confluence.test.dimagi.com and create an HTML redirect file as above. 

You do not need to add any redirect on Cloud itself.

You cannot redirect to a Cloud instance with Proxy Pass as you are doing. The redirect needs to completely forward traffic to the new URL. If you do not access Confluence directly from the URL confluencetest100.atlassian.net, then it will not work.

I hope that clarifies things.

Let me know if you continue to have any trouble.

Regards,

Shannon

[Sanjay Chinthala]

.

[Sanjay Chinthala]

.

[Shannon S]

Sanjay,

The index.html was the only thing you would do if you were completely getting rid of Confluence on your hosting server. 

The easiest thing for you to do, which I recommended earlier, is as follows:

1. On the Server Instance, navigate to Confluence Administration > General Configuration > Custom HTML (Under Look and Feel)
2. Click Edit at the bottom of the page.
3. Under At end of the HEAD, insert the following code:
   

   <meta http-equiv="Refresh" content="0; url=https://confluencetest100.atlassian.net/wiki/spaces/commcarepublic/overview" /> 

4. Click Save.

Now, if you visit your Server instance (confluencetest.dimagi.com) it will redirect to your Cloud instance. If you're not logged in, it will ask you to login again.

This does not require you to configure anything further on the server. 

For the issues you are having with Proxy Pass, can you let me know what exactly you are trying to do with that? The redirect does not require it.

Thank you for your help!

Regards,

Shannon

[Sanjay Chinthala]

.

[Shannon S]

Sanjay,

The HTML redirect I mentioned does work, as I tested your URL and I am redirected properly. I do not see any NGINX error so it might be an issue in your network.

For the ProxyPass issue, I'm really not sure what you're trying to do here. It seems that you might be trying to redirect the pages individually to Cloud, but as I mentioned previously this is not possible.

Regards,

Shannon

[Sanjay Chinthala]

.

[Shannon S]

Sanjay,

You can redirect the entire Server instance to Cloud as I have shown you, but you cannot redirect individual pages in Confluence Server to their equivalent in Cloud. Is this what you are trying to do?

Regards,

Shannon

[Sanjay Chinthala]

Hi Shannon,

most of our customers using the wiki and help sites and which will redirect relevant pages on confluence and is any way to redirect these pages to confluence cloud

is anyway to redirect wiki and help sites to relevant pages on confluence cloud?

Thanks,

Sanjay

[Shannon S]

Sanjay,

You need to use the HTML redirect no matter where you want to direct your users. Either in the header, which redirects the whole site, or on individual pages.

I hope that's clear. Let me know if you have any questions.

Regards,

Shannon

[Sanjay Chinthala]

.

[Shannon S]

Sanjay,

With the example HTML redirect I showed you, it simply redirects all traffic that arrives on that page to the new page you specify, such as https://confluencetest100.atlassian.net/wiki/spaces/commcarepublic/overview. There's absolutely no reason why would need to set up two separate redirects for wikitest.* and helptest.* You can have both point to the same index file since you want them to end up at the same place.

This topic doesn't have a lot to do with Confluence itself. I would recommend that you get in touch with your server admin regarding this if you have any further questions.

Thank you for your understanding!

Regards,

Shannon

[Sanjay Chinthala]

Hi Shannon,

can we use custom domain name to serve the cloud site?

https://jira.atlassian.com/browse/CLOUD-6999

Thanks,

Sanjay

[Shannon S]

Sanjay,

It is not possible at this time to use custom domains with Cloud. The feature request you linked is accepted for implementation but has not been completed yet. 

In order to access your Cloud site you need to navigate directly to your *.atlassian.net instance. This is the reason I said your redirect has to go there directly and cannot be a proxy.

If you have any other questions I highly recommend that you raise a ticket with the Cloud Support Team. This thread is quite long at this time and is no longer related to your original question. You're also sharing a lot of private information here which is being made public. Please make sure not to share any private information here such as your instance URLs, email addresses, etc. I've removed your email address above for your privacy.

Take care,

Shannon

[Sanjay Chinthala]

Thanks, Shannon and I have removed private data such as emails and URLs from this thread and thanks for helping a lot.

[Shannon S]

You're welcome, Sanjay! Thank you for letting me know you were able to remove that information from the thread. 

Take care, and have a pleasant rest of your week.

Regards,

Shannon