Is it feasible to do a day trip to anapolis by public transit?
No. The only reasonable way is to drive.
By public transit, no, but there are private buses that make the trip. Young Transportation (https://www.ytsonline.net) has the most flexible schedule, with morning and evening trips in both directions and weekend service. It leaves from New Carrollton metro. Dillon's (http://www.dillonbus.com/) runs commuter buses designed to take people from Annapolis to downtown DC in the morning, and back to Annapolis in the evening, which is not useful for a day trip but could be helpful if you wanted to stay overnight.
You could even take a taxi from New Carrollton Station (or Largo Town Center Station) (it might be easier than a car rental). It's probably $50 each way, especially if you want a larger car. It's 23 miles from New Carrollton and 24 from Largo.
A taxi sounds like a good idea, that wouldn't be that bad a cost for several people at all.
I recommend uber as there are not always am an sitting at these stations