Upcoming changes to AQL in Assets for Jira Service Management cloud

[#1]

Hello Atlassian Community,

We have some important updates to share regarding AQL in Assets for Jira Service Management in cloud.

We're in the process of making fundamental improvements in performance, scalability, and reliability for Assets in Jira Service Management. We’ve set ourselves some ambitious goals which will bring huge improvements for all of our customers. As a part of this effort, we’ve determined that some of the AQL features we offer today will not perform or be reliable enough in their current form - thus hindering our ability to create the Enterprise-grade platform we want to provide you. Due to this we are deprecating some features and changing the behaviour of others. Based on your feedback, we will explore returning some of these in future in a form that allows us to meet the performance and reliability standards you expect from us.

All changes listed below come into effect on or after 30 Sept 2024 and will apply to use of AQL in our user experiences and public API endpoints.

Removal of the CIDR function

The CIDR function allows filtering over IP ranges, for example

"IP Address" IN CIDR("192.0.0.0/8")

If this is a form of search you require, as an alternative we suggest using LIKE to search for IP addresses in a particular range, e.g.

"IP Address" LIKE "192.0.0"

TextArea attributes will no longer be searchable with AQL or other Assets search experiences

Not to be confused with Text attributes which will continue to be searchable as they are today. If you have information contained with a TextArea attribute that you need to search on, we recommend moving it into one or more Text attributes.

The behaviour of ordering on User, Group, Project, Bitbucket repository and Opsgenie team attributes is changing

Ordering via the Assets user experience or by using an AQL ORDER BY will no longer have the same behaviour for attributes of type User, Group, Project, Bitbucket repository and Opsgenie team. Ordering will be based on the underlying ID of the attribute value as opposed to the visible name. We understand this will likely not be the behaviour most of you expect in this situation. For most customers the new behaviour will appear as “grouping” as opposed to “ordering” as the underlying ID is not visible but values with the same ID will appear together after ordering. Ordering by all other attribute types will continue to work as they do today.

Ordering of AQL results by attributes with a maximum cardinality greater than 1 will not be supported

Ordering via the Assets user experience or by using an AQL ORDER BY will no longer be supported for attributes with a maximum cardinality configured as greater than 1. Currently very few customers use ordering on attributes of this kind and the behaviour often does not match what our customers expect. Ordering will continue to be supported on all attributes with a maximum cardinality of 1.

Greater than and less than operators will only be supported on numeric and date attribute types

Today it is possible to use the >, >=, <, <= operators on attribute types where this is typically not required. Use of these operators will be limited to Integer, Float, Date and Date time where the use case and expected behaviour is clear.

Nested use of the connectedTickets() function within inboundReferences() and outboundReferences() functions will not be supported

The connectedTickets() function will not be usable when nested within the inboundReferences() or outboundReferences() functions. This means an AQL query such as

object HAVING connectedTickets(labels is empty)

(i.e. objects connected to issues where the label field is empty) will continue to behave as it does today. However,

object HAVING inboundReferences(object having connectedTickets(labels is empty))

(i.e. objects with an inbound reference from other objects which are connected to issues where the label field is empty) will not be valid AQL.

Using dot operator(.) in the context of an import mapping will not be supported

Use of the dot operator(.) will not be possible when using AQL as a part of mapping source data to an object in an import configuration. This restriction will apply only to the use of AQL in the import mapping section of the Assets import experience as well as the import endpoints in our public REST API. In all other locations the the dot operator(.) will continue to function as it always has.

While we have assessed that most of these changes will have a low impact on the majority of our customers, we understand that for some of you this may not be the case and we apologise for any disruption this may cause you. As always please reach out to us if you have any concerns or feedback.

All the best,

Justin King
Senior Product Manager, Jira Service Management

[Masayuki Abe]

@Justin King 

Hi,

I am very pleased that Atlassian is investing in making Assets more reliable, and I am confident that Assets has the potential to become a reliable, centrally managed database. As you know, the data stored in Assets can be combined with automation rules and other products to provide a wide range of solutions for customers.

I have heard that you are working on fixing the bugs in JSDCLOUD-12246 and I look forward to hearing good news.

[Justin King]

Hi @Masayuki Abe. The work we are doing will resolve JSDCLOUD-12246. It's not a small job so its going to take a little while but its happening. Stay tuned!

[Rick Westbrock]

If this helps resolve that defect then I can live with the constraints. The only one that is a disappointment is that Text Area attributes will no longer be searchable but I can understand from a technical standpoint why that is the case.

[Justin King]

Thanks for your understanding @Rick Westbrock. Another workaround for the changes to TextArea could be linking to a Confluence page (via a URL attribute type) so you can use the search capability in Confluence. Whether this helps or not will obviously be heavily dependent on your use case. 

Would you be willing to share a bit more on the types of searches you do on TextArea's today and the problems it is solving for you?

[Rick Westbrock]

We migrated from another system to JSM and had several multi-line text fields that were migrated during that project. The object type with the Text Area attributes is Location Info and if a user wants to search in the Assets UI for which object contains a specific vendor name (e.g. a cabling company which did some work at a location) they would need to search a Text Area attribute.

At this time the other system is still up so users are still searching there so I don't know how often they might need to do the same searches in Assets. This object type has something like 500 objects and 4-5 Text Area attributes so I don't think we can surface them in Confluence for searching as that UX would not be very good.

[Andreas Krupp]

HI @Justin King ,

Could you elaborate how JSDCLOUD-12246 and the removal of these functions are related? They seem completely unrelated.

From a user perspective I think there are a couple of features that might introduce some disappointments with our customers:

• TextArea Field search not possible -> Then you might just remove that field fully because if you cannot search for keywords in a large CMDB, then this information should not be there in the first place.
• Changing the sorting order -> Why can you not keep ORDER BY for the "name/label" attribute? Being able to sort by underlying IDs is not useful at all
• Nesting of connectedTickets Function -> That is used quite a lot on big Data Center CMDBs, because you simply want to know all "Incidents" related to an "Application Group", which is not connected to any ticket but is metadata
• CIDR Function - also used in larger CMDBs quite frequently to navigate through subnets

You might think that these functions are not broadly used - and you are right - it is because the big CMDBs with lots of data inside and deep integration into all the areas of asset management, configuration management, deployment management, service management, etc.,etc. are still on Data Center and not in the cloud.

Assets is the nucleus of all we do in Jira. I understand that you have hard choices to make, but please prioritize some pressing matters:

• Limit to only connect 50 Assets to a ticket
• Import capabilities
• Mapping seamlessly Atlassian ORG Users to Imported users via Email without using a costly Automation - afterall you are on the same platform, this should be easy
• API Calls to create Asset Custom Fields including their Configuration

Cheers & thx!

andreas

[Justin King]

Hi @Rick Westbrock 

A question for you if don't mind elaborating? In the case of your location object, would there be a record of a vendor doing work at a location in the form of a linked Jira issue that could be searched on? I'm just trying to understand your workflow. Feel free to reach out to me at jking@atlassian.com or book a time to chat if you would prefer not to make that information public. 

Hey @Andreas Krupp 

Thanks for reaching out!

On the issue of JSDCLOUD-12246. This problem (regular downtime) stems from Assets not being architected correctly for the cloud. Our cloud version of Assets is architected a similar way to the Server / DC version (where it originated) and makes the assumption that updates are rolled out irregularly and that when are they are downtime is acceptable. We are completely rebuilding the cloud version of Assets to remove assumptions of this kind (we want to roll out updates to you as frequently as is needed without causing you any interruptions).

However removing the regular downtime this is only one of our goals. We also want to offer you:

• Performance and reliability guarantees for queries even when using Assets at very large scale under heavy load.
• Significantly larger, faster and more reliable imports. 
• The ability to store many more objects than you can today (10's of millions)

The reality is that the features that worked well in a Server / DC environment do not always work well in a cloud environment with the above objectives. This is why we are trimming down the feature set a little in the short term to allow us to meet these goals as quickly ass possible. After that is done we can explore bringing back capabilities that may be important to you in a way that is appropriately scalable and reliable in cloud.

I understand this might be disruptive if you are using these features and I apologise for that! We are not making any of these decisions lightly. Its a tradeoff we are making because I believe that offering you a reliable product is critical. I don't want to offer a feature only to have it be unreliable or perform below expectations. This is the situation we have put ourselves in today and we need to fix it.

To address a couple of your questions specifically: 

Ordering by User, Group, Project, Bitbucket repository and Opsgenie team

This performs without problems in a Server / DC environment because everything is running on a single machine. The Atlassian cloud product suite is a distributed system consisting of many services. In the case of Assets and users, the user information is not stored in Assets when you add a user into an attribute on an object. Instead we store the ID the user. When we want to perform operations using information like the users name or email, we make calls to the appropriate services that manage this information. The latency introduced by this means that ordering a larger number of objects by this information does not perform within expectations and can lead to timeouts. There are however ways to solve this in a cloud environment (you can obviously do this in other Atlassian products e.g. Jira). For example caching of relevant information within Assets is one approach we may take. This is not a small undertaking though so in order to meet the reliability objectives we have set sooner, we are making a short term trade off with ordering. 

TextArea Field search not possible 

While I agree the removal of search means some use cases for this field may be removed, we believe there are still many cases where a customer may wish to store large amounts of text (e.g. notes, descriptions etc) on an object even if they are not the subject of a search. In future we plan to allow rich text in this attribute bringing you the full capabilities of the Atlassian editor.

Thanks for your feedback! If you would like to chat about this further please feel free to make a time.

Justin

[Rick Westbrock]

@Justin King the Location Info objects which have the multi-line text fields are only linked to the Site object for the same location; the Site object type has core attributes like city/state/street address while the Location Info object type has more detailed information like telecom data for the DEMARC, notes from circuit installations etc.

The Location Info object are not linked to any JSM tickets, the tickets would be linked to the Site object (which has no multi-line attributes) instead.

[Justin King]

Thanks @Rick WestbrockAnd whats the rationale for splitting the Site and Location info object types up?

[Rick Westbrock]

The Site objects are updated by an automated data pipeline so we do not allow users to modify those objects. All attributes that are not part of that data pipeline are in the Location Info objects so they can be updated by users.

If the permissions model were more granular to allow read or read/write at the attribute level then we could put all of these attributes into one object type.

[Jonathan Boulgier]

@Justin King Making text area fields non-searchable broke a number of critical processes in JSM for us this morning. Removing this feature was clearly not discussed with any Atlassian customers and goes counter to Atlassian's core principles. I am extremely disappointed that Atlassian has once again made a change that harms their customers instead of helping them.

On another note, how do we include Opsgenie (now Operations in JSM) as a field in an asset schema? I do not see this anywhere so how can we sort by it if it does not exist?

[Paolo Pozzoli]

I Confirm that we need to be able to search in textarea fields since it contains different information we are getting for assets. Functionality is required.

[Justin King]

Hi @Jonathan Boulgier

I'm sorry to hear you were disrupted. We did announce that this change was coming with approximately 6 months notice on Atlassian Community, Atlassian Developer Community, and in the Atlassian changelog as is our standard practice. It sounds like in this case additional messaging may be been useful to ensure everyone recieved the news. 

I can assure you we are not making changes with the intent to harm customers and we are very conscious of avoiding changes like this as much as possible. We are doing this in service of our efforts to ensure we can offer a service that performs acceptably and is reliable at very large scale. Offering AQL over arbitrarily large amounts of rich text at scale is regrettably not something we can offer in a reliable or performant way at the present time. 

On the Opsgenie question. Do you mind clarifying what you mean by an "Opsgenie field"?

[Gideon Nolte _Jodocus_]

Hi @Justin King

I need your help understanding the changes regarding comparison of numeric values. I am wondering why IP addresses are not considered numeric values anymore. We implemented a use-case at a customer, which requires IP comparison to assign discovered devices to the corresponding network, based on the device IP. In my opinion IP addresses ARE numeric values. More specifically, they are 32bit integers with a funny representation. Why are they no longer comparable?

If I understand the reasoning correctly, you want to

[Remove the] operators on attribute types where this is typically not required [and only] [u]se these [...] where the use case and expected behaviour is clear.

In my option comparison of IP addresses is typically required - for example at the enterprise customer mentioned above - and the use and behavior of comparison operators would be clear and well defined.

After reaching out to support since one of our IP comparison process started producing errors, I was informed that IPs are handled as strings. On that note I would understand why you removed the comparison but then again, handling IPs as strings is tech debt the customer should not have to pay for. 

Suggestion: Save on storage by reducing a stored IP from 15 to 4 bytes by saving them as an integer and give back comparison for these then numeric values. Added benefit: your CIDR function will no longer be a pain in the a** and can just be substituted by gt/lt comparisons.

Seems like an overall easy win for me: What am I missing?

Greetings

Gideon