Mainly because different groups of customers will have different needs, different reasons for buying your products. By segmenting you can identify those needs by customer segment and target your marketing more effectively.
Company size. If you sell B2B then one standard technique is to segment by size and size is either by number of employees or by revenues. There are something like 1 million businesses in the UK with less than ten employees and then another 200,000 with 10-99 employees and those two groups account for well over 90% of total UK businesses. So if you plan to segment by number of employees and go after the small businesses, you will need to segment further to be able to do any effective targeting.
Geographic location. Many companies sell just to their local area, they can quite easily use local directories to gather information on local businesses or do postcode based mailshots to reach local people. The internet changes everything and it is common to find a small, local shop shipping its wears around the globe.
Demographics. Age, sex, social status etc. Many products and services sold to consumers will be most appropriate for certain sub-groups but don't stop at the obvious groups for a product, children's clothes may be mostly bought by parents but there will be other interested groups as well, do the same due diligence for your own products and services. Newspapers can be a good way of targeting specific social groups but don't assume you know what they are, check with the papers for their readership data to be sure you get it right.
Commuter types. If you wanted to sell to car drivers then posters along major roadways might be a good option; similarly users of the underground or metro are likely to look at advertising when they are desperately trying to avoid eye contact with their fellow travellers. There are many types of commuter and finding the ways your target audience is likely to travel is a very effective way of making the best of your advertising budget.
Media. Some groups are self selecting and there are many trade and profession based publications both in print and on the internet that can let you target your chosen customer group with laser accuracy. If you chose this route you will have to make sure you do your homework on your target customers needs. When they are reading their own dedicated press they will expect any advert or editorial to use the right language and be knowledgeable on the subject matter being delivered.
The chances are you will use several of these methods together or some of the many others. The main thing to remember is why you are segmenting; it is so that you can write to a specific customer group's needs and show them that your company and your products are the ones that will best address those needs.