Node.js API Pagination Support

[Ray Cooke]

Hi,

I'm trying to query the OpsGenie API from a Node.js app and I'm using the supplied Node.js library to do it. The library doesn't appear to have any support for paginated results and it appears the only way to do it from the API is using the paging section of the response which lists the next URL, however this is a long way down the call stack inside the library. Is there a recommended pattern for implementing pagination using the library or do I need to rewrite the library from scratch?

Cheers,

Ray

[Ray Cooke · Accepted solution]

Transferred from thread above:

For those that find this and want an answer, please use the below (tested and works). It's substantially different, and more generic, to what I put earlier in this thread and more importantly works, whereas the earlier content I put won't because of the async nature of the API call.

/**
 * Query the OG API for data
 * 
 * @param {*} ogAPICall The API to call, e.g. opsgenie.incident.list
 * @param {*} queryJSON The query parameters, e.g. {query: "", sort: "insertedAt", order: "desc"}
 * @param {*[]} [results=[]] Internally used. Maintains a recursively constructed array of results
 * @param {*} [startAt=0] Interally used. The offset index for paged results fetching 
 * @returns {Promise<[]>} of results
 */
async function queryAPI(ogAPICall, queryJSON, results = [], startAt = 0) {

  // Set pagination parameters
  queryJSON.offset = startAt;
  queryJSON.limit = 100;
  queryJSON.direction = "next";

  return opsGeniePagedRequest(ogAPICall, queryJSON).then(response => {
    console.log(response);

    results = results.concat(response.data);

    // If there are more results to get then get them.
    /* 
     Note: In the situation where the number of results is a whole multiple of the limit
     one wasted API call will be made. This cannot be avoided without knowing the full
     number of results, which the API response doesn't currently provide.
     */
    if (response.data.length == queryJSON.limit) {
      // Chain up another promise
      return queryAPI(results, startAt + queryJSON.limit);
    }
    return results;

  }).catch(error => {
    console.error(error);
    return results;
  })
}

/**
 * Wrap an OpsGenie library API call in a Promise so that we can handle pagination
 * 
 * @param {*} ogAPICall The Ops Genie library API call to make
 * @param {*} queryJSON The JSON object defining the query parameters
 * @returns {Promise} that the API call will be made
 */
async function opsGeniePagedRequest(ogAPICall, queryJSON) {
  return new Promise((resolve, reject) => {
    ogAPICall(queryJSON, (error, response) => {
      if (error) reject(error);
      resolve(response);
    });
  });
}

An example of using the above would be:

let incidents = await queryAPI(opsgenie.incident.list, {query: "status : open", sort: "insertedAt", order: "desc"});

[Darryl Lee]

Hi @Ray Cooke ,

This is Darryl. I am here to help. 😃

Understand that you would like to seek a way to implement the pagination on the queried results via the Node.js library.

You are correct, the fastest and most accurate way to achieve that is to find the `next URL` value in the result for the next API call.

However, you may also leverage the fields, `offset` and `limit`, while sending the API calls via the Node.js library.

Take List Alerts for example:

The example library code snippet is

var list_alert_json = {
    query : "status : open",
    offset : 0,
    limit : 10,
    sort : "alias",
    oder : "desc"
};

opsgenie.alertV2.list(list_alert_json, function (error, alerts) {
    if (error) {
        console.error(error);
    } else {
        console.log("List Alert Response");
        console.log(alerts);
    }
});

The `limit` field is defining the maximum number of items to provide in the result. Must be a positive integer value. The maximum value is 100.

The `offset` field is the start index of the result set (to apply pagination). The minimum value (and also the default value) is 0.

This means you can use this as the 1st API call:

var list_alert_json = {
query : "status : open",
offset : 0,
limit : 100,
sort : "alias",
oder : "desc"
};

While in the next API call:

var list_alert_json = {
query : "status : open",
offset : 101,
limit : 100,
sort : "alias",
oder : "desc"
};

and so on.

---

If this is not able to help you accomplish the objective, please share more details and your expectation with us to help you better.

Thank you very much for your feedback.😃

Kind regards,
Darryl Lee
Support Engineer, Atlassian

[Ray Cooke]

Thanks for the response Darryl. I saw that, but I don't see a field that tells me how many results there are going to be, so how do I know when to stop? My working assumption is that I don't do another request if I get less results back than the limit I've specified, but that doesn't cover the edge case where the number of results is a multiple of the limit. I will be making an extra call for a start index > than the total results set. At best I'll get back 0 results which will be a wasted API call. At worst I'll get an error.

Also, just checking, presumably the offset is 0 indexed, so the second page would have an offset of 100 in your example right?

Cheers,

Ray

[Ray Cooke]

So I would use something like the below (please correct me if I'm wrong), but as above, that results in a wasted API call when there is a number of results which is a whole multiple of the limit:

var incidents = [];
var listIncidentsJSON = {
  query : "status : open",
  offset : 0,
  limit : 100, // 100 is the maximum single result set OpsGenie API allows
  sort : "insertedAt",
  order : "desc"
};

do {

  await opsgenie.incident.list(listIncidentsJSON, (error, response) => {

    if (error) {
      console.error(error);
    } else {
      incidents = incidents.concat(response.data);
    }

  });

  listIncidentsJSON.offset += listIncidentsJSON.limit;

} while (response.data.length == listIncidentsJSON.limit);

console.log(incidents);

[Ray Cooke]

Note: Added "await" above since I assume this will just infinite loop otherwise.

@Darryl Lee Could you confirm on whether there's a way to avoid that edge case described above? Is there a result set total size included somewhere?

[Darryl Lee]

Hi @Ray Cooke ,

Great point!

Your understanding is correct. Sorry for the typo I made in the offset value of the 2nd API call.

This one you made is perfect:

listIncidentsJSON.offset += limit;

Indeed, without knowing the total number of the results, it will be a waste of API calls.

As I am not an expert in Node.JS, here I take the List Alert API call for example.

In the result of the 1st API call (actually it's the same in every API call), there is an object, paging, at the end of the result. Among the child fields within paging, the field, last, contains the URL for the final API call which reveals the total amount of entries in the query result by leveraging the offset value.

For example, the following result indicates the last entry of the queried result should be contained within the result by calling offest=100 while limit=100:

"paging": {

"next": "https://api.opsgenie.com/v2/alerts?limit=100&sort=createdAt&offset=100&order=desc",

"first": "https://api.opsgenie.com/v2/alerts?limit=100&sort=createdAt&offset=0&order=desc",

"last": "https://api.opsgenie.com/v2/alerts?limit=100&sort=createdAt&offset=100&order=desc"

}

I am wondering, in your code snippet, would it be possible to fetch this from the 1st API call result to define the number of API calls you should use each time?

Hope this makes sense.

Kind regards,
Darryl Lee
Support Engineer, Atlassian

[Ray Cooke]

Hi @Darryl Lee,

Understood. Yep, I guess I could write something to parse the URL on the paging block but that's pretty sub-optimal as an implementation - loads of extra wasted CPU cycles. I guess I'll have to do that in the meantime though.

Is there any way we can / how do I get either:

1st choice: Get the library implemented to deal with pagination itself, or;

2nd choice: Get a total results field of some kind added to the response object?

... or both in an ideal world I suppose, so people can pick their implementation choice?

Cheers,

Ray

[Darryl Lee]

Hi @Ray Cooke ,

Unfortunately, there is no other way to around this to my knowledge.

I can relate to where this frustration comes from and I have just created a feature request on your behalf, OPSGENIE-549. Please vote it and set yourself as a Watcher to receive the latest updates.

Thank you very much for your feedback in helping us improve Opsgenie. 😃

Kind regards,
Darryl Lee
Support Engineer, Atlassian

[Ray Cooke]

Removed original so I can repost on main thread as acceptable answer