The Vizianagaram Goldsmiths Association has asked the Union government to clarify its stand on gold buying, saying goldsmiths have been hit after Modi's remark.