Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,298,560
Community Members
 
Community Events
165
Community Groups

Node.js API Pagination Support

Hi,

I'm trying to query the OpsGenie API from a Node.js app and I'm using the supplied Node.js library to do it. The library doesn't appear to have any support for paginated results and it appears the only way to do it from the API is using the paging section of the response which lists the next URL, however this is a long way down the call stack inside the library. Is there a recommended pattern for implementing pagination using the library or do I need to rewrite the library from scratch?

Cheers,

Ray

2 answers

1 accepted

1 vote
Answer accepted

Transferred from thread above:

For those that find this and want an answer, please use the below (tested and works). It's substantially different, and more generic, to what I put earlier in this thread and more importantly works, whereas the earlier content I put won't because of the async nature of the API call.

/**
* Query the OG API for data
*
* @param {*} ogAPICall The API to call, e.g. opsgenie.incident.list
* @param {*} queryJSON The query parameters, e.g. {query: "", sort: "insertedAt", order: "desc"}
* @param {*[]} [results=[]] Internally used. Maintains a recursively constructed array of results
* @param {*} [startAt=0] Interally used. The offset index for paged results fetching
* @returns {Promise<[]>} of results
*/
async function queryAPI(ogAPICall, queryJSON, results = [], startAt = 0) {

// Set pagination parameters
queryJSON.offset = startAt;
queryJSON.limit = 100;
queryJSON.direction = "next";

return opsGeniePagedRequest(ogAPICall, queryJSON).then(response => {
console.log(response);

results = results.concat(response.data);

// If there are more results to get then get them.
/*
Note: In the situation where the number of results is a whole multiple of the limit
one wasted API call will be made. This cannot be avoided without knowing the full
number of results, which the API response doesn't currently provide.
*/
if (response.data.length == queryJSON.limit) {
// Chain up another promise
return queryAPI(results, startAt + queryJSON.limit);
}
return results;

}).catch(error => {
console.error(error);
return results;
})
}

/**
* Wrap an OpsGenie library API call in a Promise so that we can handle pagination
*
* @param {*} ogAPICall The Ops Genie library API call to make
* @param {*} queryJSON The JSON object defining the query parameters
* @returns {Promise} that the API call will be made
*/
async function opsGeniePagedRequest(ogAPICall, queryJSON) {
return new Promise((resolve, reject) => {
ogAPICall(queryJSON, (error, response) => {
if (error) reject(error);
resolve(response);
});
});
}

An example of using the above would be:

let incidents = await queryAPI(opsgenie.incident.list, {query: "status : open", sort: "insertedAt", order: "desc"});
0 votes
Darryl Lee Atlassian Team May 30, 2022

Hi @Ray Cooke ,

This is Darryl. I am here to help. 😃

Understand that you would like to seek a way to implement the pagination on the queried results via the Node.js library.

You are correct, the fastest and most accurate way to achieve that is to find the `next URL` value in the result for the next API call.

However, you may also leverage the fields, `offset` and `limit`, while sending the API calls via the Node.js library.

Take List Alerts for example:

The example library code snippet is

var list_alert_json = {
query : "status : open",
offset : 0,
limit : 10,
sort : "alias",
oder : "desc"
};

opsgenie.alertV2.list(list_alert_json, function (error, alerts) {
if (error) {
console.error(error);
} else {
console.log("List Alert Response");
console.log(alerts);
}
});

The `limit` field is defining the maximum number of items to provide in the result. Must be a positive integer value. The maximum value is 100.

The `offset` field is the start index of the result set (to apply pagination). The minimum value (and also the default value) is 0.

This means you can use this as the 1st API call:

var list_alert_json = {
query : "status : open",
offset : 0,
limit : 100,
sort : "alias",
oder : "desc"
};

While in the next API call:

var list_alert_json = {
query : "status : open",
offset : 101,
limit : 100,
sort : "alias",
oder : "desc"
};

and so on.


If this is not able to help you accomplish the objective, please share more details and your expectation with us to help you better.

Thank you very much for your feedback.😃

Kind regards,
Darryl Lee
Support Engineer, Atlassian

Thanks for the response Darryl. I saw that, but I don't see a field that tells me how many results there are going to be, so how do I know when to stop? My working assumption is that I don't do another request if I get less results back than the limit I've specified, but that doesn't cover the edge case where the number of results is a multiple of the limit. I will be making an extra call for a start index > than the total results set. At best I'll get back 0 results which will be a wasted API call. At worst I'll get an error.

Also, just checking, presumably the offset is 0 indexed, so the second page would have an offset of 100 in your example right?

Cheers,

Ray

Like Darryl Lee likes this

So I would use something like the below (please correct me if I'm wrong), but as above, that results in a wasted API call when there is a number of results which is a whole multiple of the limit:

var incidents = [];
var listIncidentsJSON = {
query : "status : open",
offset : 0,
limit : 100, // 100 is the maximum single result set OpsGenie API allows
sort : "insertedAt",
order : "desc"
};

do {

await opsgenie.incident.list(listIncidentsJSON, (error, response) => {

if (error) {
console.error(error);
} else {
incidents = incidents.concat(response.data);
}

});

listIncidentsJSON.offset += listIncidentsJSON.limit;

}
while (response.data.length == listIncidentsJSON.limit);

console.log(incidents);
Like Darryl Lee likes this

Note: Added "await" above since I assume this will just infinite loop otherwise.

@Darryl Lee Could you confirm on whether there's a way to avoid that edge case described above? Is there a result set total size included somewhere?

Darryl Lee Atlassian Team May 30, 2022

Hi @Ray Cooke ,

Great point!

Your understanding is correct. Sorry for the typo I made in the offset value of the 2nd API call.

This one you made is perfect:

listIncidentsJSON.offset += limit;

Indeed, without knowing the total number of the results, it will be a waste of API calls.

As I am not an expert in Node.JS, here I take the List Alert API call for example.

In the result of the 1st API call (actually it's the same in every API call), there is an object, paging, at the end of the result. Among the child fields within paging, the field, last, contains the URL for the final API call which reveals the total amount of entries in the query result by leveraging the offset value.

For example, the following result indicates the last entry of the queried result should be contained within the result by calling offest=100 while limit=100:

"paging": {

"next": "https://api.opsgenie.com/v2/alerts?limit=100&sort=createdAt&offset=100&order=desc",

"first": "https://api.opsgenie.com/v2/alerts?limit=100&sort=createdAt&offset=0&order=desc",

"last": "https://api.opsgenie.com/v2/alerts?limit=100&sort=createdAt&offset=100&order=desc"

}

I am wondering, in your code snippet, would it be possible to fetch this from the 1st API call result to define the number of API calls you should use each time?

Hope this makes sense.

Kind regards,
Darryl Lee
Support Engineer, Atlassian

Like Ray Cooke likes this

Hi @Darryl Lee,

Understood. Yep, I guess I could write something to parse the URL on the paging block but that's pretty sub-optimal as an implementation - loads of extra wasted CPU cycles. I guess I'll have to do that in the meantime though.

Is there any way we can / how do I get either:

1st choice: Get the library implemented to deal with pagination itself, or;

2nd choice: Get a total results field of some kind added to the response object?

... or both in an ideal world I suppose, so people can pick their implementation choice?

Cheers,

Ray

Like Darryl Lee likes this

Hi @Ray Cooke ,

Unfortunately, there is no other way to around this to my knowledge.

I can relate to where this frustration comes from and I have just created a feature request on your behalf, OPSGENIE-549. Please vote it and set yourself as a Watcher to receive the latest updates.

Thank you very much for your feedback in helping us improve Opsgenie. 😃

Kind regards,
Darryl Lee
Support Engineer, Atlassian

Removed original so I can repost on main thread as acceptable answer

Like Darryl Lee likes this

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
Community showcase
Published in Confluence

An update on Confluence Cloud customer feedback – June 2022

Hi everyone, We’re always looking at how to improve Confluence and customer feedback plays an important role in making sure we're investing in the areas that will bring the most value to the most c...

208 views 1 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you