I am having a similar problem, but no passphrase is being asked for:
$ ssh -i elasticbamboo.pk ec2-user@ec2-54-234-XXX-XXX.compute-1.amazonaws.com
The authenticity of host 'ec2-54-234-78-100.compute-1.amazonaws.com (54.234.XXX.XXX)' can't be established.
RSA key fingerprint is a3:37:bf:ac:fc:e3:28:8e:34:75:53:f5:e2:5a:de:f7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-54-234-XXX-XXX.compute-1.amazonaws.com,54.234.XXX.XXX' (RSA) to the list of known hosts.
Permission denied (publickey).