But the Borderers could count themselves lucky not to have lost by a bigger margin. The defeat was their third in a row in the league.
Berwick made a bright opening, pinning Albion back in the early stages, and a 20-yard shot from Berwick's Jay Shields appeared to be handled but penalty appeals were ignored.
Rovers replied when Craig Coyle pulled off a great double save to deny George Wallace and Stephen Reid when he pounced on the ball.
Gareth Hutchison almost put the Borderers ahead with a low shot which beat the keeper but came back off the post and was cleared. Stephen Donachy had a great chance to put Albion in front, but he pulled the shot wide after pouncing on a pass-back and rounding the advancing keeper.
Albion's Stephen Reid lobbed over on the hour mark with the goal at his mercy before the keeper dived at his feet to save.
Albion finally got the breakthrough after 70 minutes when Chris McGroaty mis-hit a sliced shot by Kevin Haynes, leaving Scott Chaplain to drill home.
Berwick were denied an equaliser when Gordon Lennon cleared off the line from Gary Hutchinson, who then saw the keeper pull off a great diving save at the death.