The visitors almost took the lead inside the first minute when a Denis McLaughlin lob from 30 yards rebounded off the woodwork.
Albion then went close themselves when Andy Sim saw his volley well saved by Gary O'Connor in the Berwick goal.
On 24 minutes Berwick took the lead when Albion keeper Jamie Ewings failed to gather an Iain Thomson corner and Fraser was on hand to fire home.
Early in the second half Albion almost grabbed the equaliser when Joe Savage saw his header cleared off the line by Tom Britton.
Berwick then had the chance to put the game beyond doubt when they were awarded a penalty as Scott Friel was sent off after handling a goal ward bound shot by McLoughlin.
However, Ewings produced an excellent save to deny Gary Wood from the resulting penalty.
This sparked the home side into life as they went in search of the equaliser.
They should have had it on 80 minutes when substitute Bryan Felvus saw his shot superbly saved by O'Connor just before Pat Walker saw his header rebound off the woodwork.
All the home side's late pressure was to no avail as Berwick held on to claim all three points.