Yup, happened to me, broke the trim, had to spend a little over a Benjamin for a replacement unit on Ebay.
On my replacement assembly, I used two needle nose screwdrivers, and lifted the trim from the back side of the trim, not from the front. It easily came apart, just do not use pliers to remove the trim. Good luck!