Postgres - Schema check for table [EVENTS] encountered an error

Jiri Pik February 26, 2018

How to troubleshoot this?

 

2018-02-26 18:06:42,072 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [EVENTS] encountered an error. Turn debug logging on for stacktrace.
2018-02-26 18:06:42,081 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [SECRETS] encountered an error. Turn debug logging on for stacktrace.
2018-02-26 18:06:42,082 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [SNAPSHOTS] encountered an error. Turn debug logging on for stacktrace.
2

1 answer

1 accepted

1 vote
Answer accepted
AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 26, 2018

Hi Jiri,

Are you using PostgreSQL? I found this in a support ticket that reported the same error: 

...these warnings are not worrying at all and won't hinder the functionality or performance of any feature in the application. We have seen it appear on other instances running on PostgreSQL, but never got reports of any performance degradation.

Are you experiencing any unexpected/undesirable behavior from Confluence? If not, the warnings may be ignored. If so, please let me know what is not working as expected so we can take a closer look.

Cheers,

Ann

Jiri Pik February 26, 2018

Yes, PostgreSql running on AWS RDS.

There was a Denial of Service attack on my confluence - which caused the PostgreSql to reject any connections - I think the attackers might have tried to insert some special values into username and password and it made PostgreSql reject all calls from Confluence. On the same server, I am running also JIRA which was unaffacted. The solution was to restart PostgreSQL

ZZZZZ is the server name. 

The same thing happened today. 

Also, sometimes when i restart the server, confluence is not starting up - again most likely database problem.

 

Are there any special PostgreSQL params in RDS i should have a look atr?

 

018-02-19 00:03:52,052 ERROR [C3POPooledConneccionPoolHanager[IdencicyToken->lbr8bmq9t1aicapcsimluk|55074Odf]-HelperThread-#1] [org.poscgresql.Driver] connect; Connection error:
rg.postgresql.util.PSQLException: Connection to ZZZZZ:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections, at org.postgresql.core.v3.ConnectionFactorylmpl.openConnectionlmpl(ConnectionFactorylmpl.java:265) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:4 9) at org.postgresql.j dbc.PgConnection.<init>(PgConnection.j ava:194) at org.postgresql.Driver.makeConnection(Driver.j ava:450) at org.postgresql.Driver.connect(Driver.java:252)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.j ava:220) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.j ava:206)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$lPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.j ava:203) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125) at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoo1AsynchronousRunner.j ava:6 96) aused by: java.net.ConnectException: Network is unreachable (connect failed) at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPIainSocketImpl.doConnect(AbstractPlainSocketImpl.j ava:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.j ava:2 0 6)
at java.net.AbstractPIainSocketImpl.connect(AbstractPIainSocketImpl.j ava:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.j ava:3 92)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.j ava:68)
at org.postgresql.core.v3.ConnectionFactorylmpl.openConnectionlmpl(ConnectionFactorylmpl.j ava:14 4)
... 13 more

Jiri Pik February 26, 2018

Could the problem be the max number of open connections? Or something along those lines - the restart of PostgreSql always solves the problem?

Jiri Pik February 26, 2018
AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 26, 2018

I am not sure why the article cannot be actioned in RDS. Is there no access to the /var/lib/psql/pg_hba.conf file in RDS? 

I use PostgreSQL myself and that (Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections) is the error I get when PostgreSQL is not started on the specified host. Please try connecting with a tool like PGAdmin to verify the host and port are reachable when the server is throwing that error.

The database connection pool in Confluence can be adjusted per this guide: Confluence slows and times out during periods of high load due to database connection pool

Jiri Pik February 26, 2018

In RDS you have no access to the underlying PostgreSQL filesystem. 

JIRA works at the moment when the Confluence is throwing the error. 

AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 26, 2018

I learn something every day: Support Amazon RDS It looks like RDS is not a supported platform for Confluence. Can you install plain vanilla PostgreSQL on an AWS Linux host just using a package manager? Presumably that would give you access to the configuration files.

Jiri Pik February 26, 2018

I found this - https://forums.aws.amazon.com/thread.jspa?messageID=515545 - i will try it - and have no idea why the problem is only with Confluence and not JIRA.

 

Thanks for all your help!

AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 26, 2018

It seems like if it was ident vs. trust Jira would have the same issue: JIRA Unable to connect to database with 'Ident authentication failed' error Also, it doesn't make sense that the authentication would intermittently work for Confluence but sometimes fail.

Jiri Pik February 26, 2018

Yes, there is something odd going on. Something like firewall or running out of connection or something. 

If I find it, I will report it here. 

AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 26, 2018

Really appreciate your offer to circle back and let us know what you find!

Jiri Pik February 26, 2018

On the login page to Confluence, what happens if I enter some magic username or some magic password -> do you do any preprocessing, like escaping?

would it be possible that an attacker injects a special SQL command to username which forces the PostgreSql to reject all connection to the confluence?

JIRA has not been under any attack recently, just Confluence.

AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 26, 2018

I researched and asked my colleagues, we wanted to share this, Confluence Security Overview and Advisories:

Confluence interacts with the database through the Hibernate Object-Relational mapper. Database queries are generated using standard APIs for parameter replacement rather than string concatenation. As such, Confluence is highly resistant to SQL injection attacks.

One theory: If the attackers got your database specs somehow they could have used up all the configured connections, leaving none for Confluence.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events