I think it’s less about that we are looking at people, but more about the length/intensity of eyecontact in general.
Other countries might consider it impolite to inspect a passing stranger’s choice of clothing or keeping eyelock during a conversation.
Continuous eyecontact combined with a neutral expression might appear condescending or judgemental to foreigners, but that is just my interpretation.
If you walk through the streets and you think passer-bys are staring at you, they probably are not. There is a cultural difference to be aware of, the german stare.
Here it is completely normal to look at people while passing them, or even look out from a coffee shop or bus at passing people. It might signal Neugierde - interest, or might just be general Aufmerksamkeit - attention of their surroundings. Think of it like a cat acknowleging that you have entered the room by giving you a quick glance.
Please don’t feel like you are being singled out - you will see that everyone gets looked at :)
Sway 1.12, released last week, added support for capturing individual windows.