To clarify what people mean when they say “privileged” does not mean that you were not abandoned or abused (and I am sorry that happened to you). “Privileged” does not in any way mean your life was easy. What it means is that your skin color is not one of the reasons your life was difficult. It means that your life would have been even more difficult if you were black and living in those same circumstances.
For example, when you applied for a job, people didn’t throw out your resume because of your name. When you went to a store, security didn’t follow you around because of your skin color. Cops didn’t pull you over because of your skin color.
Also, no one is suggesting that because you’re white, you’re a bigot. All people are suggesting is recognizing that although your life was difficult for a multitude of reasons, skin color wasn’t one those reasons. And for non-white people, their skin color typically holds them back even if they were privileged in other ways (e.g., if they’re wealthy, grow up in a stable family).
There are many types of privilege (economic, racial, gender, sexuality, having a stable home environment). Having one type of privilege puts you one step ahead. It does not mean that you also have those other types of privileges. And, if you don’t have those other types of privileges, your life may indeed be very difficult. I am sorry you faced those other difficulties, and hope this clarifies what privilege means.