Absolutely it's worth it. Boston's Logan airport is just the other side of Boston Harbour from downtown.
The quickest way downtown is public transport. At the airport, take the free shuttle bus to the Airport "T" (subway) station, from where you should take the "T" to State, which is right downtown. Children under 11 go free.
A lot of the main sites are within walking distance from State station (which is right under the Old State House): Faneuil Hall, Boston Common, State House, Beacon Hill. There's a tourist information office right beside State station.