I had about 10k in CPP income, nothing else and yet I had to pay 2400 to the CRA. Don't I get a personal exemption? I did have US income of 120k.

Without knowing your entire tax situation, it's tough to advise why you are having to pay this amount.

Are you filing as a Non-Resident? 

Yes, I filed as a non resident because I live in the US