What Determines Breast Size? There are many factors to consider when deciding what is truly the average breast size. What determines breast size can vary for many reasons such as genetics, weight, age, health, exercise, and pregnancies. Studies have found that heavy coffee drinkers were associated with smaller breasts. Also, young girls start developing breasts around the age of 8 years old and this may continue till mid-twenties. Therefore, you must consider both sides of your family.