cryptsetup luksRemoveKey <device>

You will be prompted for the passphrase you wish to remove and then for any one of the remaining passphrases for authentication.