You can also right-click on the Peek button and uncheck the Peek at Desktop option. To disable the Windows Preview feature in Windows 7, right-click on the taskbar and click open Properties.

It is update time for those using PowerToys on Windows 10 and 11 ... it does add previewing the content of archive files through Peek, .reg files can be set to open by default in Registry Preview ...