Hi, either could be the issue - handle itself wears and there is a spring within the lock body that could be affected.
If you have another similar type handle / lock in another door, swap around the lock body & see if the same issue is seen.
If you can, take off the handles and check them - there should be a metal wire / spring behind it. Spray a bit of WD40 on that, clean out any gunk in there. You can also take out the lock body, open that up (careful that the spring doesn't pop out), again, some WD40 in there.
Alternatively, might be time to replace the lock / handles - just be aware of changing the locks / handles as the cut-out within the door may not suit a different type of handle / lock body. Change like for like & it should be ok
Also, depending on the door, you may need to get longer or thicker screws as the original holes in the door, from the handle, may not hold the screws securely when replacing.