It actually affects attack range, damage collision, max stamina and stamina regeneration, you can go to secret paths if you're small enough, maximum encumberment and some windy paths affects you differently if you're too heavy or too small to get pushed back. So yeah, it affects a lot actually...