On a quiet weekend in movie theaters, “Mean Girls” repeated atop the box office with $11.7 million in ticket sales, according to studio estimates Sunday, while a handful of awards contenders sought to make an impact ahead of Oscar nominations Tuesday.