I wrote a checkpoint file to my scratch directory; it wasn there when the job finished. What happenned?
If it was written to your main scratch directory ( /scratch/user, where user is your userid, look in the chk_archive directory. Checkpoint files can be large, and people were not cleaning them up; much time was spent copying them around. Now, part of what LSF does when it copies files back up to the login host is to put any *.chk files it finds in a user’s main scratch directory into the chk_archive directory, which isn’t copied down to the execution host. If you need to use a *.chk file with a job, just move it out of the chk_archive directory before you submit the job.