Yes, Jews worshipped at the Temple. But they distinguished themselves by treating their God as being invisible. They had to do so to retain their culture after the Temple's destruction. They sharpened their monotheism to make their God portable. Jewish self-understanding evolved, and the scriptures were edited in hindsight. So yes, there are complications. But the symbols in the Temple (the Holy of Holies, etc) were for Jews' benefit. Eventually, they learned not to treat those symbols as idols.