In the late 1800's, the physicists at the University of Vienna were moved into a horrible building at 3 Turkenstrasse. Here's a quotation from Maria Rentetzi's article (in Endeavour 28 [March, 2004]):
"The Physicalisches Kabinett and Stefan's Physics Institute moved into the four-story building located in Turkenstrasse 3, a short side street crossing Wahringerstrasse. A 'very primitive, converted apartment house,' as Lise Meitner described it...."
Another description of the building is in Ruth Lewin Sime's 1997 biography of Lise Meitner (Lise Meitner: a Life in Physics [University of California Press]): "The physics institute was on the Turkenstrasse ..., on the same block as the institutes for pharmaceutical chemistry and medicinal chemistry. ... A photographer's studio and a coffee house stood on either side of Turkenstrasse 3 .... Originally the structure had been a small apartment house, already run-down when the university purchased it as a temporary building in 1875 (a permanent physics building opened in 1913). Its entrance reminded Lise Meitner of the door to a hen house. 'I often thought "If a fire breaks out here, very few of us will get out alive."'"
The coffee house mentioned above was at 12 Wahringer Strasse, at the corner of Turkenstrasse. I've included here two photos of that site from August, 2008. Given the inadequacy of the physics quarters, the physicists seem to have sought refuge in the little cafe at the corner. Here's Rentetzi again:
"The same architectural absurdity led the physicists, working at Turkenstrasse, to the charming coffee house at the corner of their laboratory and contributed to the confluence of scientific disciplines and discussions. Of that 'idyllic time', the physicist Karl Przibram later recalled with nostalgia that 'the young generation of physicists can hardly imagine the passion of the debates, echoed in those days, particularly in the above mentioned coffee house.'  The 'above mentioned coffee house' was a cafe at the corner of Wahringerstrasse and Turkenstrasse, one of the distinct Viennese cultural spots, which literally housed the young physicists of the Institute." ['19' is a reference to Przibram's 1959 'Erinnerungen ...' in Beitrage zur Physik und Chemie des 20. Jahrhunderts, ed. O. R. Frisch et al]
While in Vienna, I bought a book called Kaffeehaus Album: 1860 - 1930 by H. Seemann and C. Lunzer (Album Verlag, 2000). On p. 58, they include a 1925 picture of Cafe Astoria's interior, which was located at 12 Wahringerstrasse. So, Cafe Astoria appears to have been the physics hotspot of Vienna.
Update (Sept. 12): Maria Rentetzi has published a book with Columbia University Press entitled Trafficking Materials and Gendered Experimental Practices. She has made the full contents of the book available on-line. Chapter 3 of her book ('Gender, Science, and the City') has more information on physics in Vienna in the early 20th Century and includes a photo of several physicists in their cafe.