I am trying to clean up labels and at the bottom of the pages I can add and seemingly delete labels.

However, if I do it at he bottom of the page rather than the edit labels way, labels that have no pages linked still exist in my all labels list. This is the only way I've noticed this problem occurring. If I remove labels the edit labels direction, this is not a problem and the label will disappear if not attached to a space or page.
How can I remove labels that have no attached space or page in Confluence?
