Insiders at agencies tasked with delivering Victoria’s infrastructure program claim waste and political priorities have created a miserable place to work.