Union minister says although he wants to stay away from controversies, if his party continued to be targeted, a reply will be given at an appropriate time