Isn there a loss of torque associated with fitting lightweight pulleys, since the flywheel effect is lost?
No. In early days it was common to fit large heavy flywheels to increase torque, which to a certain extent does work, but it is important to look at the context of the application. A heavy weight acting as a flywheel on the crankshaft has a greater resistance to a change in rotational velocity, which is termed inertia. The term “resistance to change in rotational velocity” applies to both acceleration and deceleration, so a heavy flywheel or pulley does oppose deceleration in situations such as towing caravans up a hill, but it will also oppose fast acceleration.