Both teams arrived at 3Arena with everything to play for after last week's stalemate in the first leg in Norway.
A more open game looked to be in store this time around, as Ole Saeter tested Warner Hahn before Jusef Erabi and Nahir Besara tried to land the first blow for the hosts, but both were denied by Sander Tangvik.
From then on, Hammarby continued to dominate in the first half, with Besara continuing to look lively while chances came and went for Montader Madjed, but for the third time, a 45-minute period ended goalless.
The Bajen began the second half as they ended the first, though Erabi still couldn't find a way past Tangvik.
Just before the hour mark, the hosts were punished for their lack of clinical finishing when Rosenborg finally broke the deadlock - Selnaes tried to force the ball goalwards and Dino Islamovic was on hand to divert it past Hahn and into the back of the net.
That was the only effort on target of the three total shots the visitors managed in the second half, while Hammarby peppered the Rosenborg goal with 10 strikes after the break. However, a lack of accuracy was their downfall, asthey also produced just one effort on goal.
The home sie threw everything at Rosenborg in the final minutes, but neither Shaquille Pinas nor Sebastian Tounekti could find a way through, before they coughed up their best chance with the last action of the match.
Pinas swept in a perfectly timed cross from the left side to the unmarked Erabi in the middle of the penalty area, but somehow the centre-forward sent his header over the bar.
The final whistle seconds later signalled the end of Hammarby's European adventure, while Rosenborg move on to the UECL Playoff Round.