I have a friend who's sister had a baby when she was very young and gave her up for adoption as you did twenty odd years ago. She always thought about her baby and wondered how she was etc. She got a letter from her daugher through the adoption agency a couple of years ago and she was absolutely thrilled. Her daughter's parents had always told her she was adopted and were very supportive of her contacting her birth mother. They all get on so well, it's just a fantastic reunion to have happened. I'm sure it mightn't always be the case but I think if the mother or siblings are seeking you out, it's something they want dearly to meet with you. You have to remember in 99% of cases years ago it must have been so heartbreaking for a woman to give up her baby but that was what they had to do.
Go with your heart, weigh it all up, you can only do whatever you think is right. The very best of luck with whatever you decide to do.