When hovering over the card, click the pencil icon and click "edit labels". There should be a small "check" on the label that is showing. Click on the "check" to deselect it. This should make the label disappear from the card.
Additionally, when adding a label to a card, you can do so by hovering over the card and tapping the corresponding number key. ie: 1 for yellow. If you tap 1 again, it will remove the label.