What is the difference between the above mentioned formats and what purpose do they serve?
For different purposes different icon formats can be used. For example, if you are developing a software application or a GUI application for windows, you will need .ico icons whereas .gif icons can be used in web applications or any other application that can be accessed through a browser. A vector icon can be stretched to a huge extent and can be used to create high quality print outs for, use in offline advertisements, brochures etc.