How newton used to calculate the force of gravitation between two objects before value of G was known?
Newton was a beast. It is true that he did this. Basically Galileo calculated the force of gravity on Earth based on many observations, so that number was known before Newton appeared. Back then, the distance between the Earth and the moon was approximately known, as was the diameter of the Earth and the diameter of the moon. Based on the moon’s distance and speed relative to the Earth, Newton found the force of gravitation between the Earth and the moon. He somehow saw that there was a relationship, and even though his numbers were imprecise, he saw that it was approximately the inverse of the distance between the centers of the objects squared. It was basically the application of the most current math and science of the day, combined with an enormous intuitive leap.