Yes! This bothered me as well, until I recently encountered Sean Carroll's philosophy of "Poetic Naturalism":
1. There are many ways of talking about the world.
2. All good ways of talking must be consistent with one another and with the world.
3. Our purposes in the moment determine the best way of talking.
One way of talking about the Game of Life simulation running in my other browser tab is as a bunch of electrons bouncing around in my computer's CPU. Another way of talking about it is as a cellular automaton obeying Conway's rules. And they're consistent with one another; e.g., if I stop the electrons by shutting down the computer, I expect the automaton to stop running.
In retrospect, it's pretty obvious. But it must not have been _too_ obvious, because it presents a viewpoint that isn't quite physicalism and isn't quite dualism, and people have been arguing back and forth about that for a long time.
Sean Carroll, The Big Picture