You must assign your variable to the new altered string, because replace doesn't update your variable:
var description = ", you are in a room.";
description = description.replace('', req.session.player);
Moreover, if you want to replace all occurrences of '<\?player>' instead of only the first one, then use a regular expression with g (global) flag:
var description = ", you are in a room.";
description = description.replace(/<\?player>/g, req.session.player);
For full information, read https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace. Some quotes:
Returns a new string with some or all matches of a pattern replaced by a replacement.
This method does not change the String object it is called on. It simply returns a new string.
To perform a global search and replace, either include the g switch in the regular expression
