I need to specify a different SSH port but TortoiseCVS won let me?
The reason is because CVSNT, which TortoiseCVS relies on for all CVS actions, does not allow specifying the port for using the SSH protocol in the CVSROOT like other protocols. To setup TortoiseCVS to use a different SSH port go to TortoiseCVS Preferences → SSH → SSH parameters and set the value to -P portnumber, or create a PuTTY session and change the SSH configuration settings accordingly.