This letter presents a new filtering technique for interferometric synthetic aperture radar (InSAR) phase images. Traditional local denoising algorithms all suffer from the drawback of removing texture detail information. In contrast, several nonlocal methods have attained good performance in InSAR applications. However, these methods only take radiometric similarity. The patch-based locally optimal Wiener filter (PLOW) utilizes both geometrically and radiometrically similar patch information by clustering analysis and nonlocal filtering; thus, it can better balance between detail preservation and denoising. Nevertheless, PLOW itself is not fitted for InSAR. In this letter, we modify and improve the original algorithm while considering the coherence coefficient and the characteristics of InSAR. This new method provides better filtering results, but with a higher computational cost. Moreover, we introduce the box dimension in fractal geometry as a new index. Experiments on simulated and real data demonstrate that this algorithm outperforms traditional denoising methods.