Can I run Simple Home Budget as well as keep my budget files on the USB thumb drive?
Yes, you can. Simple Home Budget is very small standalone application with the size little above 1M (your smallest USB drive probably around 256M). Budget files will not grow more than 1M for couple years of extensive use. Please follow recommended procedure to safely remove USB drive to make sure all writing processes are finished.