The observation itself is not the referent, is it? Observation is a process. Isn't the "referent" you are referring to here actually the "object" observed, and by "the referents are false" you mean "the conclusions are false"?
You cannot have a conceptual referent without observing the referent at some point. It's not either/or. It's both.
(Thinking it can be either/or is probably the main fallacy in your entire approach.)
Even if you are abstracting from abstractions, which were abstracted from other abstractions, and so on, if you trace those down to the premise level, you will find direct observation of some sort.