A goal three minutes into injury time by Montrose striker Duncan McLean gave the visitors all three points.
It was the home side who had a lot of early possession, but they failed to make any clear-cut chances in front of goal.
Very much against the run of play Montrose took the lead as Neil Stephen met a Martin Wood corner after 12 minutes to head the ball home.
As a result of the goal, Rovers were forced to make a change as keeper Chris Fahey fell awkwardly whilst challenging Stephen for the ball and was replaced by Mark Peat.
James Mercer almost got Rovers back on terms after 18 minutes, but his header was somehow cleared off the line. Then again on the half-hour the big striker rose above the Montrose defence, but his header flew just over.
The visitors didn't trouble the Albion goal for most of the first half but could have gone in at half-time two goals up when McLean saw a lob fall just wide of the post.
In the second half the home side pressed hard for the equaliser and went close following long-range shots from Wayne Gordon and Jered Stirling. They eventually did get back on level terms when a Stirling long ball was latched on to by Lee Wilson and he made no mistake in firing the ball home.
This sprung the visitors into life, and they went close when substitute Kerr Dodds had a shot well saved by Peat. However, they were almost caught at the back as Wayne Gordon latched on to a Stephen mishit clearance but James Butter did well to deny the striker.
In the end both sides were going all out for the three points and Albion's Chris Silvestro had a shot well saved by Butter before McLean gave Montrose the victory with a last-gasp winner.