Hipchat native OSX client won't connect to private hipchat server

[Chris Monkman]

We've got a number of Macs running OSX sierra (10.12.5) with the latest version of hipchat client (4.29.0.732) and none of them can access our private server instance of hipchat. 

A check through the logs gives it's only hint as 'Ignoring unexpected request in OAuth2Flow' which after this appears, it gives up and starts trying again...
Ad infinitum 

I've gone through all of the checks for SSL certs, open ports and clearing any current cookies and pretty much every other method to fix it that I could find.

For the record - it works in browser, we've got it working in iOS/Android apps and I've personally got it working just fine on my own Mac (running the same versions as above, but I did have it working on OSX 10.11 El Capitan before I upgraded to Sierra) so any advice/help would be appreciated.

[Bruno Raymundo]

Hi, Chris!

Mac is stricter than other platform regarding redirections... This seems to be the case here. You can test bypassing any proxies you have and connect to the server directly, temporarily, in order to check if this is the case. If you have a load balancer or reverse proxy, please add a valid cert to HipChat as well and configure your proxy to use port 443 across the whole chain of connections.

Regards,

Bruno

[Chris Monkman]

Hi Bruno,

I've double checked and whilst some of the machine affected were running proxies others weren't. A good example of example of this is I personnally use an iMac in targeted display mode as a second monitor for my rMBP, and use connection sharing to pass the ethernet connection. On the iMac which is a fresh and clean install of OSX Sierra, I can't get hipchat to work, yet without changing a setting, hipchat authenticates and works just fine on the rMBP plugged into it.

As for the cert being valid, it's using a wildcard cert set to expire next year running off TLS 1.2 encryption so should be working fine.

As for connecting to the server directly, I've tried using the server direct address (we've got it hosted on an AWS instance) and that results in the same issues - also checked about 443 being used across all connections which they are as a matter of company policy (this is both for direct and proxied connections).

I have dug further into logs and have found an number of references to TLS being Enabled, then pending but nothing further until TIC TCP Conn Cancel which gets repeated a few times. But other than that there's nothing else much that would hint at what the issue could be.

[Bruno Raymundo]

Hi, Chris!

Is your private HipChat instance hosted on AWS? If so, please check if you have any Elastic Load Balancer settings; we've seen cases where the ELB listener was set to 443 via HTTP instead of HTTPS, which was causing this problem.

If it's hosted on a local server, check for any load balancer or reverse proxies as well.

Keep us posted!

Regards,

Bruno

[Chris Monkman]

Hi Bruno,

I've been told by our ops team that the settings are correct on the AWS instance as listed above. It did however get me thinking so I've setup a transparent proxy to listen to the traffic going to and from the server.

One of the things I've spotted is that the it stops right when it's expecting a response through the URL '/users/authorize?<various logging strings>' which is returning a 302 http header response rather than the usual 200. It should however still work off a 302 I would have thought unless it's ignoring all non 200 responses?

[Bruno Raymundo]

Hi, Chris!

Listening to the traffic was a great idea. So, the 302 responde means that the target resource was found, but is not on the expected location. While the server, when throwing this response should generate a location field containing the new location, the clients may or may not use the field value for automatic redirection.

I wonder what you'd see if you tried listening to the same traffic when using the web client; it's likely that the way the web client handles the redirection differs from the way the Mac client does. Feel free to raise a ticket with our support team so we can analyze the issue in detail, checking the logs as well and proceeding to either find a solution, a workaround or identify a new bug.

Thanks!

[Chris Monkman]

I'm aware of what a 302 is - looking further into it, the URI it's trying to use (to run a get call to /users/authorize) is a http request but gets a https response hence the 302.

So by the look of things it's not getting the response it expects when making the users/authorise get call and on further inspection there's some really odd behavour - the /users/authorise call its getting redirected to /login (along with the same query string items as the initial /users/authorize call) and after the redirect thats where it's stopping.

(It's also worth noting that the 302 location is being returned as http not https, yet the redirect body puts the link as https which is also rather confusing).

All in all it sort of looks like the app isn't making the right call to the right login page and is thus getting stuck waiting on a response (even though it got one, just not one it expected).

Could it be that the login calls in the app/server have changed between versions meaning that the app is trying to make a call to the wrong address?

I've also spotted that part of the querystring is 'redirect_uri=hipchat://hipchat.com/authorized&state=<id number>' which seems wrong to me as it should be redirecting to our servers address not to hipchat.com but I might be reading too much into that.

[Chris Monkman]

I still haven't been able to resolve this one - but I have found a further clue after looking into it again now that 4.30.x has been released.

When fishing through the logs I've looked out the working instance and re-logged back in to our team server - everything worked fine as was expected. I've then logged into a cloud account on hipchat.com on a mac that can't log in and gotten all the way through with that too.

Finally I tried logging into our team server again on a non working mac (which got stuck again) and checked the logs.

Whats become obvious is that the oAuth2Flow is getting a different authState back from the server compaired to those that work.

Specifically;- Failed log line

2017-08-31 15:35:52:367+0100  [Login] Deciding policy based on oAuth2Flow(host: "https://hipchat.mycompanydomain.co.uk", authState: 25591382, wasRetried: false): Optional("http://hipchat.mycompanydomain.co.uk/login?...#...")

Working log line

2017-08-31 15:52:56:347+0100  [Login] Allowing request by url prefix in oAuth2Flow(host: "https://hipchat.mycompanydomain.co.uk", authState: 28369112, wasRetried: false) to Optional("https://hipchat.mycompanydomain.co.uk/login?...#...")

Which implies that the authState of 25591382 is not working/triggering next step/out right wrong vs 28369112 which works just fine (both for our team server and the hipchat cloud server).

- Edited - Never mind just rechecked the working cloud login and the authState is different. So much for that idea.

Any ideas what would cause this/how to resolve it would be grately appreciated.