Paul Tait's early strike spread confidence through a Rovers team not used to winning and the timing of Giuliano Grazioli's second goal in first-half injury time, could not have been better for boss Ray Graydon.
Bury player-manager Andy Preece lifted his side with a late substitute's appearance, but Pawel Abbot's injury time goal was no more than a consolation.
Tait ran on to Grazioli's pass to open the scoring after seven minutes with a fierce right-foot finish from just inside the box.
It was one of several moments of loose Bury defending which cost them dear.
A first half of few chances developed and it was into stoppage time when Rovers extended their lead through another defensive slip.
Bury's Colin Woodthorpe went to ground as he was challenged by Grazioli for a long ball from Tait and the nippy Rovers frontman was left with only Glyn Garner to beat.
Grazioli did that confidently by lobbing the keeper as he came out.
The second half brought chances at both ends, with Vitalijs Astafjevs going close for Rovers and Scott Howie having to arch his back to tip over a header from Danny Swailes.
But the visitors only really came to life after manager Preece introduced himself as a 67th minute substitute for Martyn Forrest.
Preece inspired a late rally that saw Abbott capitalise on a strong run from substitute Dave Nugent to net from close range in the first minute of added time.
There were three more minutes left, but they were most notable for Bury assistant boss Graham Barrow being banished from the dug-out by referee Paul Armstrong for showing dissent at a throw-in.
The result was a relief to Graydon, who has had to rebuild the Rovers team this summer