You say, "we might not realize that we are taking a perspective-dependent view when we believe we're taking a perspective-invariant one." But the content of any viewpoint already varies from the thing itself, because it's a model or a map rather than the terrain. That's the subjectivity that's built into understanding X as opposed to being X.

Object permamence could be a matter of tracking rather than understanding. But if it entails understanding the general concept of physicality, it's still an afforance that's more pragmatic than a case of pure objectivity. How does object-permanence sit with quantum mechanics or with a more cosmic perspective in which so-called objects are fleeting?

The ball is permenent relative to a frame of reference. A person who lives for thousands of years might have a different conception of objects. And is the ball really the same from one moment to the next, or is that assumption pragmatic, meaning it's useful to get what you want from the environment? "Pragmatic" here means partly subjective.

Our concept of objects likely evolved to enable us to survive by exploiting patterns in our environment. Whether that innate concept reveals the ultimate truth is another matter.