403 ONLY IN GOLANG, curl and Postman work fine

[Jon Lasley]

Hi all,

Hitting any endpoint with basic auth (API token as password) from my golang app returns a 403;  however, the same endpoint with the exact same basic auth succeeds in both Postman and curl. I am very confused, as this only recently started happening to the golang app. I have tried adding `Accept` and `User-Agent` headers to no avail. I am at a complete loss here.

I have also verified with a Postman mock server that the Authorization header on the golang request is identical to the header on both the postman and curl requests.

EDIT: 

It is of note that the golang app worked up until about a week ago, then I started receiving the 403 errors.

[Eugene Zhang]

@Jon Lasley were you able to resolve the issue mentioned in the ticket? Running into it exactly the same as you when having to authenticate with basic auth in Golang app although curl, postman all worked fine.

Tried to follow suggestions by @Andy Heinzer but still no success even with the api token approach

Nvm, got it work with fixing on url

[Andy Heinzer]

Hi Jon,

I see that you are using basic auth in order to make REST API calls to Confluence Cloud.  It sounds like you are using the API Token in the same manner that a password would be used.  At one time, the API tokens could be used in the same manner as a password, however that is not supported any longer.   Please be aware that Atlassian has deprecated the use of passwords in basic auth as well.  More details are in Deprecation notice - Basic authentication with passwords.  I suspect what is happening is that our Cloud service is outright rejecting these call because it believes you're trying to use a password.

That said, it is still possible to use an API token with basic auth. We just need to make sure that instead of using the token like you would a password, you need to use an authorization header for your REST API calls.  Please see Basic Auth for REST APIs, it explains that you should be first creating a string in the format of

user@example.com:APItoken

and then base64 encode that string.  The encoded string then has be put into an authorization header.  The Jira version of the Basic Auth for REST API doc has better examples of how to do this encoding properly across operating systems (Windows, Linux, Mac).  But an example of how this call would look in curl would be

curl -D- \ 
-X GET \ 
-H "Authorization: Basic <your_encoded_string>" \ 
-H "Content-Type: application/json" \ 
"https://<your-domain.atlassian.net>/wiki/rest/api/space"

Try this instead and let me know if this helps.

Andy

[Jon Lasley]

Andy,

That is the format that I'm doing, the base64 encoding happens naturally during basic auth. The header gets encoded to "Authorization: Basic <b64 encoded>". The Postman, curl, and Golang authorization headers are the same. Whether I set basic auth and inspect the Authorization header, or base64 encode and set it manually, it produces the same result.

example:

curl -u "test@test.com:1234567" \ 
"https://circleci.atlassian.net/wiki/rest/api/content?type=page&spaceKey=~42831160" \
-v

# output

GET /wiki/rest/api/content\?type\=page\&spaceKey\=\~42831160 HTTP/1.1
> Host: circleci.atlassian.net
> Authorization: Basic dGVzdEB0ZXN0LmNvbToxMjM0NQ==
> User-Agent: curl/7.71.1
> Accept: */*

The same base64 encoded string as an Authorization header works in Postman and curl, just not from the Golang app.

In an effort to be thorough, I did the base64 encoding anyway and was met with the same response.

[Andy Heinzer]

Thanks for that information. If the base64 encoded string works in curl and postman, but not in golang, then there must be something different within that framework to account for this.  I am not familiar with using golang, so I'm not certain if this might be something to do with say the characters found in the base64 string, or something more along the lines of how the request being made utilizes headers.

If the header is not set correctly for that framework, it could explain the behavior here.  Confluence and Jira can permit anonymous REST API calls to some endpoints, and if the page or issue is accessible anonymously, then they are returned.  But if the endpoint requires permissions to see, then the authentication has to work to get back results.

I'm curious to learn more details about how this call is being made in golang.  Are you using a library such as https://github.com/Virtomize/confluence-go-api to make the REST API call?  Or is there some other method?  We have documented examples in a few languages such as node.js, java, python, php, etc.  See the Examples section in https://developer.atlassian.com/cloud/confluence/rest/api-group-content/#api-wiki-rest-api-content-post

You can notice from the headers of those examples that node.js uses the node-fetch library, while python uses the requests library, and PHP is using the unirest library.  It might be needed to leverage another library for Golang here in order to make sure that the requests you are trying to make are being made in a consistent manner in which we know the header is being formatted properly.  If the header is formatted incorrectly, or not recognized the response from Confluence would be essentially the same as if you had no authorization header at all.

[Jon Lasley]

I'm not using any library, just using the `net/http` Golang package. It's of note that the requests were working about 2 weeks ago, but only recently have they started hitting issues.

I've tried with a barebones request as well as a request with Accept headers. GET requests work fine, only the POST is throwing the 403.

[Andy Heinzer]

The 403 error is specifically thrown when the request is forbidden. So for example, on the POST /wiki/rest/api/content 

Returned if;

• The space that the content is being created in does not exist.
• If the requesting user does not have permission to create content in it.

The GET requests don't always fairly test this.  Since if the page is publicly accessible where even anonymous users can view the page, then it doesn't matter if the authorization is failing during the call, results can be returned (if the page is public and not restricted).

I'm still leaning towards believing that there is something off with one or more of the headers supplied (Authorization, Content-Type, Accept are the three I expect to see for a POST to that example endpoint). 

I tried to decode your authorization header here, but I found the output to be just slightly different than the presented password in your example.

Example

test@test.com:1234567

Decoded results

test@test.com:12345

This might not be valid, since I'm sure this is just an example and not your real credentials.

Could you share with us an example of the Golang code here being used to form the request including headers? 

I can see your previous curl example, but it appears to have been using the -u switch. That kind of call used to work, but with the deprecation of passwords in basic auth (link in previous reply), those requests are not expected to be able authenticate when you explicitly use that kind of switch that accepts a username/user account (even with valid password or token).  Of course please obscure your encoded token should you include an example.

[Jon Lasley]

Sure, here is the code in the project:

func (p *Page) PostPage(user, key string) *http.Response {
    client := http.Client{}
    body := p.marshalBody()
    url := fmt.Sprintf("https://company.atlassian.net/wiki/rest/api/content/%s", p.ID)
    req, err := http.NewRequest("PUT", url, bytes.NewBuffer(body))
    
    if err != nil {
        log.Error("error creating post page request :%s", err.Error())
        return nil
    }

    req.SetBasicAuth(user, key)
    req.Header.Add("Content-Type", "application/json")

    res, err := client.Do(req)
    if err != nil {
        log.Error("error doing post page request :%s", err.Error())
        return nil
    }
    if res.StatusCode >= 400 {
        log.Error("error posting: %v", res.StatusCode)
        log.Infof("basic auth pass: %s", key)
    } else {
        log.Debug("response code from post: %v", res.StatusCode)
    }

    return res
}

Like I said, this code worked about a month ago and no changes have been made.

I have been running this test to verify things:

func TestAuth(t *testing.T) {
    body := struct {
        Type string `json:"type,omitempty"`
        Title string `json:"title,omitempty"`
        Space struct {
            Key string `json:"key,omitempty"`
        } `json:"space,omitempty"`
        Body struct {
            Storage struct {
                Value string `json:"value,omitempty"`
                Representation string `json:"representation,omitempty"`
            } `json:"storage,omitempty"`
        } `json:"body,omitempty"`
    }{}
    body.Body.Storage.Value = "This is a test"
    body.Body.Storage.Representation = "storage"
    body.Space.Key = "~42831160"
    body.Type = "page"
    body.Title = "test 3"

    b, err := json.Marshal(body)
    if err != nil {
        t.Error(err.Error())
    }
    t.Log(string(b))
    req, _ := http.NewRequest("POST", "https://company.atlassian.net/wiki/rest/api/content/", bytes.NewBuffer(b))

    // debug: b64 encode myself instead of letting basic auth do it
    enc := base64.RawStdEncoding.EncodeToString([]byte("test@test.com:password"))

    req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", enc))
    // req.SetBasicAuth("test@test.com", "password")

    auth := req.Header.Get("Authorization")
    t.Logf("Auth: %s", auth)

    client := &http.Client{}
    res, err := client.Do(req)
    if err != nil {
        t.Log(err)
    }

    b, _ = io.ReadAll(res.Body)
    defer res.Body.Close()

    t.Log(string(b))
}

[Jon Lasley]

I am using a POST in the test instead of a PUT, because the PUT JSON is much more complicated and I don't want to have to mess with it every time. 

[Andy Heinzer]

Thanks for that info.  I noticed that one of you headers has this syntax:

req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", enc))

However when using Basic auth, we're not expected to use the Bearer parameter.  That is only used when using OAuth token grant as a means of authenticating.  In the Basic Auth for REST APIs it indicates that we need to be used the Basic parameter instead.

Try changing that and see if the call works.  I could see that one syntax difference being a possible reason that Jira appears to be rejecting your authentication attempt here.