As you actually "contributed" the entire 20k to your RRSP, you should be entitled to use that entire amount for your Home Buyer's Plan withdrawal. CRA will have record of you contributing the full 20k, but carry forward 5k of "unused contributions".
