Abraham's Sons
Abraham's two sons, Ishmael and Isaac, are described as symbols for those born of flesh and those born of promise. The numbers point at a book with two generations that exemplify the pattern.
Galatians 4:21-5:1
Count
2+2 = 4 = Numbers
Numbers
The book of Numbers is about two generations of Hebrews. The first are those who were of age when they left Egypt. They were counted, from 19 years of age and up, in the first census, shortly after leaving Egypt. The second generation are those who were under age when they left Egypt or born during the 40 years in the wilderness. They also were counted, 19 years and up, in a second census, shortly before crossing the Jordan River.
Paul's symbolic use of Abraham's two sons works really well with these two generations. One example is the question of inheritance. Ishmael was not able to inherit, but Isaac was. In the book of Numbers the first generation died in the wilderness when they were unwilling to go into the promise land, but the second generation would eventually inherit the promise.
Ishmael's mother was Egyptian while Isaac's was Hebrew. The two generations that left Egypt were both Hebrew by descent, but the first generation, having grown up in Egypt, were Egyptian in their cultural orientation and thinking. That was their single biggest problem since Yahvah was out to free them from thinking and living like slaves, but they were unable to shed Egypt. The second generation, though, was able to adjust to the new normal and obey when called to go fight for their future. They were free, like Isaac.
Paul also says the Ishmael type persecute the Isaac type. In the case of the two generations in Numbers, the first generation doomed the second to 40 years in the wilderness when they would not take Canaan, which could have been taken and settled in a matter of years. In this sense the Ishmael generation "persecuted" the Isaac generation.