I really recommend hiking up Mt. Esja. It's right by the city limits and even a part of the greater Reykjavík area. You can get there easily by bus and it only takes a short while, around 30 minutes or so, to drive there from the centre. If you don't have the time or the means to travel far out of Reykjavík, for something like a glacier hike, this is the perfect option.
This basic but really scenic and beautiful hike is very popular amongst the locals as a day-time activity, since you can begin and end this kind of trip in the city itself. After a short walk up the marked trail, you come to a selection of going either left or right, with the left path being shorter but steeper. Most people hike up to 'Steinn' - a stone labelled as a monument with a fantastic view of Mosfellsbær and the surrounding countryside. From there, you can hike all the way up to the top if you wish.
Just be careful if hiking outside of the summer months, since accidents happen during winter due to avalanches and sharp winds.
Have fun on your travels!