Madrid is a great city to visit. I am not sure what the negatives you have been told, but it could be around pickpockets. Like any other big city you need to take the usual precautions, but in general, Madrid is a safe city.
Madrid has great museums to visit, El Prado, Reina Sofia, Thyssen. If you friend, is interested or even not interested, the Bernabeu is worth visiting.
There are plenty of bus tours of the city, which can be easily found around the city center. Day trips to Toledo or Segovia are worth doing, but would recommend one day trip to Toledo. But you should try to do it on your own rather than go with a tour. Going with a tour is expensive.