Do you guys think people just go circle around the Ka'ba and then come back?
Pilgrims are divided into groups(their nationality) and are given a specific place to stay during their time there. Sheikhs and interpreters from these countries are present there or with Hujjajs to lead the pilgrims and guide/help them through the Hajj season. Each group is given the flag of the country they're representing so that they don't get lost, is easy to identify if somethings happens, help the pilgrims understand the Hajj in their native tongue, be with their fellow countrymen and etc
PS keep in mind that majority of the pilgrims are not arabic speakers and are doing Hajj for the first time.