Quietly subversive, this New York exhibition illuminates the friction and experimentation behind the paintings’ polish