Memory formation starts early in life, but the earliest memories that most people can recall typically date back to when they were around 3 to 4 years old. This phenomenon, known as childhood amnesia, occurs because the brain structures involved in memory, such as the hippocampus, are still maturing during the early years.