I'm creating an automation and when a ticket is moved to In Progress with 0 Points I would like to warn the user of this, so they take action to estimate the ticket.
It would be helpful to have a popup window as an action so we include a message in the window indicating the fact and user just dismiss the message.
Automation doesn't currently let you pop up a modal window for the user to work with on certain actions.
However, this sounds like a case for a workflow validator. Validators check that any input made to the transition is valid before the transition is performed. Input can include that gathered from the user on the transition's screen. You could add a validator that checks your story points field to make sure it has a non-0 value. When the user tries to transition the issue, they will be prompted to set the points, or else they won't be able to transition the issue. More reading about workflows and validators is available here if you need it.