Religious conference centre accused of ‘promoting racial prejudice’ by allowing the event to take place