Firstly, no we would not be lesser in numbers, I highly doubt the percentage of somalis in Somalia with a non somali mother is above 1%, let alone 5%.
Secondly, Somali is an ethnicity. Its not a tribe or social class that explicitly states one must be of paternal somali descent to be one. It's a gene pool, a people, and that includes women too. A half somali as we can clearly see would not be the same as a full somali. You might doubt this as most somali men who want to breed out do so let me give you an example.
There are afars who are of somali paternal descent. I can't remember which clans they mostly come from but these people don't claim to be somali nor do we claim them. They're culturally afar, speak afar and have mostly afar DNA. If being a somali was about paternal descent then this wouldn't be the case.
This is a common trope belief a lot of somalis have and should be debunked.