Savage was only on the park a matter of minutes when he received a throughball by Paul McGrillen, before taking a touch and driving the ball low past Mark Peat in the Rovers goals.
It was the home side that took the lead after only eight minutes following some neat play between Wayne Gordon and Graeme McCaul, before the latter released Lee Wilson who curved a shot home.
Stenhousemuir keeper Willy McCulloch should probably have done better as he got a good hand to the ball at his near post.
Rovers almost doubled their lead after 15 minutes when Wilson chipped a ball through for Marc McKenzie but the bounce wasn't kind to the youngste,r who could only produce a rather weak effort.
Two minutes later Gordon broke down the right and played a dangerous ball across the face of goal only to see Wilson just fail to get on the end of the pass.
It was all Albion at this stage but the Warriors strikers McGrillen and David McInally were causing the home defence all sorts of problems.
The combination almost got Stenny back on level terms as, following a neat one-two with McGrillen, a McInally drive flew just past.
Ten minutes into the second half Stenhousemuir were back on level terms when substitute Savage gathered the ball from McGrillen and fired home.
It was all Stenny at this point and they were piling on a lot of pressure on the home defence and went close through a Tommy Sinclair drive from long range.
Rovers, however, tried to hit them on the break and went close themselves through long-range efforts from McKenzie and then Chris Silvestro, before a Gordon overhead kick was well saved by McCulloch.
In the dying minutes Tommy Sinclair should have given the visitors all three points as McGrillen put him through on goal but he saw his shot drop agonisingly wide of the post.