Why wont system allow me to claim full $5000 of tuition transfer from one child but it will from the other one?

$5000 is the maximum that can be transferred, but the actual amount that you have available to transfer may be reduced if part of your child's tuition is being used to reduce their taxes payable.