Single Sign On - Correct way to exchange a refresh token for /api/Party data?

Hi - so, if I use Single Sign On to authenticate a user, I get a refresh token, which I then exchange for an access token - here's the screenshot from online documentation:

And then I'm using the access_token to get data from the /api/party endpoint - no problem - except, if I test the SSO with the manager account, and I call /api/party, I get lots of results - so I'm wondering, how should I correctly use the access_token (and userName, I'm guessing) to get Party data for the authenticated user only? For instance, if I authenticate as manager, I will get userName=manager here, and then I want to call /api/Party and get only one result - the Party record for manager.

I need to make sure that this works for any user that authenticates with SSO. So, I thought of going through CsWebUser?UserId={userName}, and then calling api/Party/{ID from the CsWebUser call}, but I'm not positive that a less privileged user would be able to access the CsWebUser endpoint. This would need to work for any user that authenticates through the SSO.

Thanks!