When I allocate a hash size of 512 MB, the task manager shows that each of the Rybka processes is using 512 MB. Is this correct?
No – this is a bug (or odd feature) in the task manager. There is only one hash table, which is shared between the Rybka processes. In other words, you can allocate roughly half of your RAM for the hash table without overflowing your available RAM, despite the task manager’s indications to the contrary.