The Victorian premier admits criminals infiltrated some government work sites but still rejects calls for a royal commission into corruption on building sites.