ROC and correlation analysis

The plots below illustrate the performance of the metrics for all scenes analyzed together and for each scene separately.

auc@25, auc@50, auc@75 is the area under the ROC assuming that more than 25%, 50% or 75% of observers mark the region.

rho is Spearman's non-parametric coefficient of correlation. The correlation is computed only for the regions that were selected by more than 5% and less than 95% of observers, as explained in the paper.

Hover the mouse over the test image to see the reference image.

Hover the mouse over the result of a metric to see the visualization of false positives and false negatives. False positives are marked with red, false negatives with green and true positives with blue. The threshold was selected so that for all scenes the true-postitive rate is equal 75% for the case where at least 50% of observers mark the distortion.

Group: all

With-reference

Metric auc@25 auc@50 auc@75 rho AD 0.783 0.834 0.892 0.294 SSIM 0.825 0.873 0.916 0.373 MS-SSIM 0.859 0.871 0.87 0.371 HDR-VDP-2 0.88 0.891 0.904 0.387 sCIE-Lab 0.832 0.867 0.896 0.355 sCorrel 0.756 0.782 0.82 0.194

No-reference

Metric auc@25 auc@50 auc@75 rho AD 0.787 0.853 0.901 0.391 SSIM 0.815 0.869 0.907 0.411 MS-SSIM 0.824 0.844 0.863 0.404 HDR-VDP-2 0.862 0.87 0.887 0.415 sCIE-Lab 0.817 0.854 0.889 0.423 sCorrel 0.728 0.801 0.851 0.338

apartment - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.948 NaN NaN 0.363 SSIM 0.925 NaN NaN 0.354 MS-SSIM 0.893 NaN NaN 0.321 HDR-VDP-2 0.725 NaN NaN -0.013 sCIE-Lab 0.97 NaN NaN 0.421 sCorrel 0.332 NaN NaN -0.042	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.71 0.776 NaN 0.23 SSIM 0.669 0.892 NaN 0.187 MS-SSIM 0.652 0.933 NaN 0.103 HDR-VDP-2 0.803 0.887 NaN 0.107 sCIE-Lab 0.744 0.753 NaN 0.26 sCorrel 0.341 0.253 NaN -0.078	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
cg figures - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.94 0.972 0.985 0.582 SSIM 0.933 0.96 0.975 0.553 MS-SSIM 0.969 0.979 0.982 0.64 HDR-VDP-2 0.981 0.978 0.978 0.67 sCIE-Lab 0.929 0.958 0.972 0.558 sCorrel 0.859 0.869 0.886 0.13	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.881 0.954 0.979 0.622 SSIM 0.853 0.945 0.975 0.575 MS-SSIM 0.896 0.969 0.98 0.653 HDR-VDP-2 0.92 0.98 0.978 0.651 sCIE-Lab 0.869 0.945 0.972 0.624 sCorrel 0.84 0.857 0.877 0.289	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
disney - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.789 0.853 0.901 0.389 SSIM 0.847 0.91 0.966 0.437 MS-SSIM 0.918 0.943 0.948 0.471 HDR-VDP-2 0.96 0.959 0.981 0.63 sCIE-Lab 0.853 0.907 0.944 0.41 sCorrel 0.752 0.794 0.814 0.253	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.809 0.845 0.893 0.497 SSIM 0.87 0.911 0.963 0.545 MS-SSIM 0.918 0.938 0.955 0.589 HDR-VDP-2 0.958 0.963 0.979 0.674 sCIE-Lab 0.869 0.901 0.94 0.531 sCorrel 0.735 0.784 0.821 0.318	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
kitchen - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.667 0.755 0.816 0.332 SSIM 0.668 0.746 0.812 0.336 MS-SSIM 0.635 0.689 0.736 0.291 HDR-VDP-2 0.727 0.764 0.816 0.324 sCIE-Lab 0.666 0.744 0.8 0.348 sCorrel 0.544 0.588 0.646 0.021	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.665 0.712 0.806 0.295 SSIM 0.668 0.716 0.811 0.296 MS-SSIM 0.622 0.653 0.728 0.201 HDR-VDP-2 0.749 0.763 0.823 0.338 sCIE-Lab 0.659 0.7 0.792 0.275 sCorrel 0.531 0.572 0.64 0.183	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
red kitchen - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.538 0.605 0.67 0.134 SSIM 0.617 0.681 0.731 0.209 MS-SSIM 0.575 0.628 0.714 0.2 HDR-VDP-2 0.822 0.835 0.884 0.267 sCIE-Lab 0.495 0.557 0.597 0.101 sCorrel 0.696 0.707 0.699 0.276	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.536 0.567 0.664 0.004 SSIM 0.618 0.652 0.74 0.065 MS-SSIM 0.579 0.61 0.678 0.015 HDR-VDP-2 0.83 0.843 0.869 0.491 sCIE-Lab 0.493 0.526 0.618 0.008 sCorrel 0.695 0.727 0.738 0.137	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
sponza above tree - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.882 0.896 0.963 0.497 SSIM 0.949 0.95 0.96 0.55 MS-SSIM 0.983 0.973 0.916 0.631 HDR-VDP-2 0.973 0.966 0.968 0.567 sCIE-Lab 0.978 0.963 0.963 0.588 sCorrel 0.874 0.88 0.789 0.429	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.831 0.654 NaN 0.469 SSIM 0.904 0.851 NaN 0.579 MS-SSIM 0.953 0.859 NaN 0.679 HDR-VDP-2 0.956 0.919 NaN 0.666 sCIE-Lab 0.947 0.888 NaN 0.655 sCorrel 0.823 0.826 NaN 0.452	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
sponza arches - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.719 0.808 0.808 0.255 SSIM 0.785 0.876 0.889 0.357 MS-SSIM 0.827 0.859 0.744 0.285 HDR-VDP-2 0.827 0.859 0.873 0.302 sCIE-Lab 0.848 0.892 0.877 0.346 sCorrel 0.737 0.81 0.755 0.23	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.725 0.851 0.947 0.235 SSIM 0.799 0.9 0.962 0.311 MS-SSIM 0.831 0.878 0.931 0.306 HDR-VDP-2 0.847 0.86 0.955 0.309 sCIE-Lab 0.851 0.897 0.952 0.353 sCorrel 0.745 0.812 0.801 0.318	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
sponza atrium - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.799 0.88 0.964 0.48 SSIM 0.877 0.947 0.984 0.586 MS-SSIM 0.958 0.963 0.97 0.474 HDR-VDP-2 0.949 0.95 0.969 0.447 sCIE-Lab 0.974 0.979 0.982 0.65 sCorrel 0.782 0.851 0.915 0.44	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.772 0.898 NaN 0.286 SSIM 0.856 0.96 NaN 0.305 MS-SSIM 0.951 0.963 NaN 0.342 HDR-VDP-2 0.955 0.972 NaN 0.337 sCIE-Lab 0.958 0.977 NaN 0.361 sCorrel 0.738 0.87 NaN 0.26	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
sponza tree shadows - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.793 0.827 0.851 0.435 SSIM 0.855 0.884 0.919 0.482 MS-SSIM 0.959 0.92 0.854 0.409 HDR-VDP-2 0.896 0.881 0.936 0.205 sCIE-Lab 0.929 0.916 0.905 0.499 sCorrel 0.818 0.822 0.813 0.343	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.773 0.772 NaN 0.405 SSIM 0.835 0.873 NaN 0.526 MS-SSIM 0.858 0.83 NaN 0.524 HDR-VDP-2 0.868 0.891 NaN 0.479 sCIE-Lab 0.863 0.854 NaN 0.495 sCorrel 0.763 0.756 NaN 0.461	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map
sponza trees - test image	With-reference Metric auc@25 auc@50 auc@75 rho AD 0.948 0.995 NaN 0.498 SSIM 0.977 0.995 NaN 0.539 MS-SSIM 0.986 0.992 NaN 0.587 HDR-VDP-2 0.974 0.988 NaN 0.564 sCIE-Lab 0.968 0.995 NaN 0.534 sCorrel 0.809 0.815 NaN 0.428	No-reference Metric auc@25 auc@50 auc@75 rho AD 0.8 NaN NaN 0.043 SSIM 0.904 NaN NaN 0.072 MS-SSIM 0.787 NaN NaN 0.134 HDR-VDP-2 0.735 NaN NaN 0.14 sCIE-Lab 0.794 NaN NaN 0.068 sCorrel 0.479 NaN NaN 0.119	Mean observer data - with reference experiment AD - distortion map SSIM - distortion map MS-SSIM - distortion map HDR-VDP-2 - distortion map sCIE-Lab - distortion map sCorrel - distortion map