bizarre color space conversion problem

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

bizarre color space conversion problem

Sarah Goslee
I am getting unexpected results when converting from RGB to Lab. This
is clearly some kind of configuration problem, but I cannot find it. I
have three linux workstations, both fully updated (except all are
running R 3.6.0 instead of 3.6.1, because that's the latest Fedora 30
binary). I'm running R --vanilla from the command line. On one
workstation, it functions correctly; on the other two the conversion
malfunctions as below.

I found it when my scico color palettes were wildly wrong, but it
isn't a scico package problem.
https://twitter.com/numberwright/status/1151601407436439554

After spending far too much time poking at it, I'm still curious as to
what might be going on. It's got to be my computer rather than R, but
I do want to figure out what, in case it has consequences for other
calculations.

Here's my sample code:

red.in <- rep("red", 20)

red.rgb   <- t(col2rgb(red.in, alpha = 0)/255)
red.lab   <- convertColor(red.rgb, from = "sRGB", to = "Lab")
convertColor(red.lab, from = "Lab", to = "sRGB")

#-#-#

Output on computer A, as expected:

> sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora 30 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.6.0
>
> red.in <- rep("red", 20)
>
> red.rgb   <- t(col2rgb(red.in, alpha = 0)/255)
> red.lab   <- convertColor(red.rgb, from = "sRGB", to = "Lab")
> convertColor(red.lab, from = "Lab", to = "sRGB")
      [,1] [,2] [,3]
 [1,]    1    0    0
 [2,]    1    0    0
 [3,]    1    0    0
 [4,]    1    0    0
 [5,]    1    0    0
 [6,]    1    0    0
 [7,]    1    0    0
 [8,]    1    0    0
 [9,]    1    0    0
[10,]    1    0    0
[11,]    1    0    0
[12,]    1    0    0
[13,]    1    0    0
[14,]    1    0    0
[15,]    1    0    0
[16,]    1    0    0
[17,]    1    0    0
[18,]    1    0    0
[19,]    1    0    0
[20,]    1    0    0

#-#-#

Output on computer B (and also computer C!)

> sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora 30 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.6.0
>
> red.in <- rep("red", 20)
>
> red.rgb   <- t(col2rgb(red.in, alpha = 0)/255)
> red.lab   <- convertColor(red.rgb, from = "sRGB", to = "Lab")
> convertColor(red.lab, from = "Lab", to = "sRGB")
      [,1]   [,2] [,3]
 [1,]    1 0.0000    1
 [2,]    1 0.0000    1
 [3,]    1 0.0000    1
 [4,]    1 0.0000    1
 [5,]    0 0.5472    0
 [6,]    0 0.5472    0
 [7,]    0 0.5472    0
 [8,]    0 0.5472    0
 [9,]    1 0.0000    1
[10,]    1 0.0000    1
[11,]    1 0.0000    1
[12,]    1 0.0000    1
[13,]    0 0.5472    0
[14,]    0 0.5472    0
[15,]    0 0.5472    0
[16,]    0 0.5472    0
[17,]    1 0.0000    0
[18,]    1 0.0000    0
[19,]    1 0.0000    0
[20,]    1 0.0000    0

#-#-#

I've gotten as far as locating the problem in this line from
grDevices::convertColor()

    xyz <- from$toXYZ(color, from.ref.white)


Packages are up to date on all computers, although this doesn't
involve any contributed packages.

Thoughts?

--
Sarah Goslee (she/her)
http://www.numberwright.com

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: bizarre color space conversion problem

Ivan Krylov
This is, indeed, bizzare!

On Thu, 18 Jul 2019 10:35:57 -0400
Sarah Goslee <[hidden email]> wrote:

> I've gotten as far as locating the problem in this line from
> grDevices::convertColor()
>
>     xyz <- from$toXYZ(color, from.ref.white)

So if you manually feed the arguments:

Lab <- structure(c(53.484183785248, 53.484183785248, 53.484183785248,
53.484183785248, 53.484183785248, 53.484183785248, 53.484183785248,
53.484183785248, 53.484183785248, 53.484183785248, 53.484183785248,
53.484183785248, 53.484183785248, 53.484183785248, 53.484183785248,
53.484183785248, 53.484183785248, 53.484183785248, 53.484183785248,
53.484183785248, 80.0102716478952, 80.0102716478952, 80.0102716478952,
80.0102716478952, 80.0102716478952, 80.0102716478952, 80.0102716478952,
80.0102716478952, 80.0102716478952, 80.0102716478952, 80.0102716478952,
80.0102716478952, 80.0102716478952, 80.0102716478952, 80.0102716478952,
80.0102716478952, 80.0102716478952, 80.0102716478952, 80.0102716478952,
80.0102716478952, 67.3840746825475, 67.3840746825475, 67.3840746825475,
67.3840746825475, 67.3840746825475, 67.3840746825475, 67.3840746825475,
67.3840746825475, 67.3840746825475, 67.3840746825475, 67.3840746825475,
67.3840746825475, 67.3840746825475, 67.3840746825475, 67.3840746825475,
67.3840746825475, 67.3840746825475, 67.3840746825475, 67.3840746825475,
67.3840746825475), .Dim = c(20L, 3L), .Dimnames = list(NULL, c("L",
"a", "b")))
white <- c(x = 0.953205712549377, 1, y = 1.08538438164692)

to the function:

grDevices::colorspaces[['Lab']][['toXYZ']](Lab, white)

Do you get the same result on all three computers?

If you debugonce(grDevices::colorspaces[['Lab']][['toXYZ']]), then
try to convertColor(red.lab, from = "Lab", to = "sRGB"), are the
arguments Lab and white the same on all three computers?

--
Best regards,
Ivan

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: bizarre color space conversion problem

Sarah Goslee
I didn't include enough detail, despite the length of my original
email. The problem is in the conversion from RGB to Lab. I converted
it back to RGB because most of us are more familiar with that. Below
are the intermediate steps.

And to make it even more bizarre, it fails with eight or more colors,
but works correctly with seven or fewer.

Ivan's suggestion works fine; the Lab to RGB conversion is okay. I
tried to figure out how colorspaces() works, but couldn't get it to
agree with convertColors(), although they're both reversible using the
same function.

> white <- c(x = 0.953205712549377, 1, y = 1.08538438164692)

> grDevices::convertColor(c(1, 0, 0), from = "sRGB", to = "Lab")
            L        a        b
[1,] 53.48418 80.01027 67.38407
>
> grDevices::colorspaces[['Lab']][['fromXYZ']](c(1, 0, 0), white)
      L       a       b
  0.000 439.086   0.000

#-#-#

Here's a more complete example.

red.in <- rep("red", 8)

red.rgb   <- t(col2rgb(red.in, alpha = 0)/255)
red.rgb

red.lab   <- convertColor(red.rgb, from = "sRGB", to = "Lab")
red.lab

convertColor(red.lab, from = "Lab", to = "sRGB")

#

> red.in <- rep("red", 8)
>
> red.rgb   <- t(col2rgb(red.in, alpha = 0)/255)
> red.rgb
     red green blue
[1,]   1     0    0
[2,]   1     0    0
[3,]   1     0    0
[4,]   1     0    0
[5,]   1     0    0
[6,]   1     0    0
[7,]   1     0    0
[8,]   1     0    0
>
> red.lab   <- convertColor(red.rgb, from = "sRGB", to = "Lab")
> red.lab
           L         a        b
[1,] 97.1495 -21.36677 94.42044
[2,] 97.1495 -21.36677 94.42044
[3,] 97.1495 -21.36677 94.42044
[4,] 97.1495 -21.36677 94.42044
[5,]  0.0000   0.00000  0.00000
[6,]  0.0000   0.00000  0.00000
[7,]  0.0000   0.00000  0.00000
[8,]  0.0000   0.00000  0.00000
>
> convertColor(red.lab, from = "Lab", to = "sRGB")
     [,1]   [,2] [,3]
[1,]    1 0.0000    1
[2,]    1 0.0000    1
[3,]    1 0.0000    1
[4,]    1 0.0000    1
[5,]    0 0.5472    0
[6,]    0 0.5472    0
[7,]    0 0.5472    0
[8,]    0 0.5472    0

#


red.in <- rep("red", 7)

red.rgb   <- t(col2rgb(red.in, alpha = 0)/255)
red.rgb

red.lab   <- convertColor(red.rgb, from = "sRGB", to = "Lab")
red.lab

convertColor(red.lab, from = "Lab", to = "sRGB")

> red.in <- rep("red", 7)
>
> red.rgb   <- t(col2rgb(red.in, alpha = 0)/255)
> red.rgb
     red green blue
[1,]   1     0    0
[2,]   1     0    0
[3,]   1     0    0
[4,]   1     0    0
[5,]   1     0    0
[6,]   1     0    0
[7,]   1     0    0
>
> red.lab   <- convertColor(red.rgb, from = "sRGB", to = "Lab")
> red.lab
            L        a        b
[1,] 53.48418 80.01027 67.38407
[2,] 53.48418 80.01027 67.38407
[3,] 53.48418 80.01027 67.38407
[4,] 53.48418 80.01027 67.38407
[5,] 53.48418 80.01027 67.38407
[6,] 53.48418 80.01027 67.38407
[7,] 53.48418 80.01027 67.38407
>
> convertColor(red.lab, from = "Lab", to = "sRGB")
     [,1] [,2] [,3]
[1,]    1    0    0
[2,]    1    0    0
[3,]    1    0    0
[4,]    1    0    0
[5,]    1    0    0
[6,]    1    0    0
[7,]    1    0    0


On Thu, Jul 18, 2019 at 11:02 AM Ivan Krylov <[hidden email]> wrote:

>
> This is, indeed, bizzare!
>
> On Thu, 18 Jul 2019 10:35:57 -0400
> Sarah Goslee <[hidden email]> wrote:
>
> > I've gotten as far as locating the problem in this line from
> > grDevices::convertColor()
> >
> >     xyz <- from$toXYZ(color, from.ref.white)
>
> So if you manually feed the arguments:
>
> Lab <- structure(c(53.484183785248, 53.484183785248, 53.484183785248,
> 53.484183785248, 53.484183785248, 53.484183785248, 53.484183785248,
> 53.484183785248, 53.484183785248, 53.484183785248, 53.484183785248,
> 53.484183785248, 53.484183785248, 53.484183785248, 53.484183785248,
> 53.484183785248, 53.484183785248, 53.484183785248, 53.484183785248,
> 53.484183785248, 80.0102716478952, 80.0102716478952, 80.0102716478952,
> 80.0102716478952, 80.0102716478952, 80.0102716478952, 80.0102716478952,
> 80.0102716478952, 80.0102716478952, 80.0102716478952, 80.0102716478952,
> 80.0102716478952, 80.0102716478952, 80.0102716478952, 80.0102716478952,
> 80.0102716478952, 80.0102716478952, 80.0102716478952, 80.0102716478952,
> 80.0102716478952, 67.3840746825475, 67.3840746825475, 67.3840746825475,
> 67.3840746825475, 67.3840746825475, 67.3840746825475, 67.3840746825475,
> 67.3840746825475, 67.3840746825475, 67.3840746825475, 67.3840746825475,
> 67.3840746825475, 67.3840746825475, 67.3840746825475, 67.3840746825475,
> 67.3840746825475, 67.3840746825475, 67.3840746825475, 67.3840746825475,
> 67.3840746825475), .Dim = c(20L, 3L), .Dimnames = list(NULL, c("L",
> "a", "b")))
> white <- c(x = 0.953205712549377, 1, y = 1.08538438164692)
>
> to the function:
>
> grDevices::colorspaces[['Lab']][['toXYZ']](Lab, white)
>
> Do you get the same result on all three computers?
>
> If you debugonce(grDevices::colorspaces[['Lab']][['toXYZ']]), then
> try to convertColor(red.lab, from = "Lab", to = "sRGB"), are the
> arguments Lab and white the same on all three computers?
>
> --
> Best regards,
> Ivan

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: bizarre color space conversion problem

Ivan Krylov
On Thu, 18 Jul 2019 11:50:17 -0400
Sarah Goslee <[hidden email]> wrote:

> The problem is in the conversion from RGB to Lab.

Hmm. Assuming defaults and skipping all checks, convertColor(red.rgb,
from = "sRGB", to = "Lab") amounts to the following:

red.rgb <- t(col2rgb(rep('red',8), alpha = 0)/255)
# let's hope this implementation detail wasn't changed in 3.6.1
white.point <- grDevices:::c2to3(grDevices:::white.points[,'D65'])
(red.xyz <- grDevices::colorspaces$sRGB$toXYZ(red.rgb, white.point))[1,]
# [1] 0.4168213 0.2149235 0.0195385
(red.lab <- grDevices::colorspaces$Lab$fromXYZ(red.xyz,white.point))[1,] #        L        a        b
# 53.48418 80.01027 67.38407
convertColor(red.rgb, from = "sRGB", to = "Lab")[1,]
#        L        a        b
# 53.48418 80.01027 67.38407

Does the manual way result in a different output? If so, do the
differences appear in red.xyz or red.lab?

grDevices::colorspaces$sRGB$toXYZ is pretty simple, just a matrix
product.  (I hope there are no differences in
as.list(environment(grDevices::colorspaces$sRGB$toXYZ))[c('dogamma','M')]
on different computers - that would be strange.)
grDevices::colorspaces$Lab$fromXYZ looks more complicated but is just a
bunch of vectorised element-wise algebra, so tracing the intermediate
results should also be possible.

> And to make it even more bizarre, it fails with eight or more colors,
> but works correctly with seven or fewer.

This is just a wild guess, but could /usr/lib64/R/lib/libRblas.so per
chance be a symlink to an OpenBLAS copy? Could any of the computers
exhibiting the bizarre behaviour be equipped with an AVX-512-capable
CPU?

--
Best regards,
Ivan

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: bizarre color space conversion problem

Sarah Goslee
I'll address all of your questions below, but starting with this:

> Could any of the computers
exhibiting the bizarre behaviour be equipped with an AVX-512-capable
CPU?

Yes. Both computers that are giving the bizarre results have Intel
i9-7900X CPUs; the X-series apparently is AVX-512-capable.

That would be an incredibly disturbing cause... But I can't find
anything else that differs. They're using the same BLAS.

I'm not even remotely a hardware expert: if the difference is due to
changes in the instruction set, I assume that has potential
consequences for other things, and I just happened to spot it in this
particular case because it's visualization-based? (Yikes.)

I really appreciate your suggestions, by the way. Thanks!


Here's a complete rundown:



As it says in my first email (but way at the bottom), I'd already
gotten as far as locating the problem in this line from
grDevices::convertColor()

    xyz <- from$toXYZ(color, from.ref.white)

Your test code confirms that as the source of the problem:


> red.rgb <- t(col2rgb(rep('red',8), alpha = 0)/255)
> # let's hope this implementation detail wasn't changed in 3.6.1
> white.point <- grDevices:::c2to3(grDevices:::white.points[,'D65'])
> (red.xyz <- grDevices::colorspaces$sRGB$toXYZ(red.rgb, white.point))[1,]
[1] 0.7733981 0.9280769 0.1383974
> # [1] 0.4168213 0.2149235 0.0195385
> (red.lab <- grDevices::colorspaces$Lab$fromXYZ(red.xyz,white.point))[1,]
        L         a         b
 97.14950 -21.36677  94.42044
> #        L        a        b
> # 53.48418 80.01027 67.38407
> convertColor(red.rgb, from = "sRGB", to = "Lab")[1,]
        L         a         b
 97.14950 -21.36677  94.42044
> #        L        a        b
> # 53.48418 80.01027 67.38407
>
>
>
>
>
> red.rgb
     red green blue
[1,]   1     0    0
[2,]   1     0    0
[3,]   1     0    0
[4,]   1     0    0
[5,]   1     0    0
[6,]   1     0    0
[7,]   1     0    0
[8,]   1     0    0
> white.point
        x                   y
0.9532057 1.0000000 1.0853844
> red.xyz
          [,1]      [,2]      [,3]
[1,] 0.7733981 0.9280769 0.1383974
[2,] 0.7733981 0.9280769 0.1383974
[3,] 0.7733981 0.9280769 0.1383974
[4,] 0.7733981 0.9280769 0.1383974
[5,] 0.0000000 0.0000000 0.0000000
[6,] 0.0000000 0.0000000 0.0000000
[7,] 0.0000000 0.0000000 0.0000000
[8,] 0.0000000 0.0000000 0.0000000
> red.lab
           L         a        b
[1,] 97.1495 -21.36677 94.42044
[2,] 97.1495 -21.36677 94.42044
[3,] 97.1495 -21.36677 94.42044
[4,] 97.1495 -21.36677 94.42044
[5,]  0.0000   0.00000  0.00000
[6,]  0.0000   0.00000  0.00000
[7,]  0.0000   0.00000  0.00000
[8,]  0.0000   0.00000  0.00000





grDevices::colorspaces$sRGB$toXYZ is pretty simple, just a matrix
product.  (I hope there are no differences in
as.list(environment(grDevices::colorspaces$sRGB$toXYZ))[c('dogamma','M')]


No, that's identical on working and nonworking computers. (Whew!)

A: working

Installed Packages
blas.x86_64                                 3.8.0-12.fc30
           @updates
liblas.x86_64                               1.8.1-1.fc30
           @fedora
openblas.x86_64                             0.3.6-2.fc30
           @updates
openblas-Rblas.x86_64                       0.3.6-2.fc30
           @updates
openblas-openmp.x86_64                      0.3.6-2.fc30
           @updates
openblas-serial.x86_64                      0.3.6-2.fc30
           @updates
openblas-srpm-macros.noarch                 2-5.fc30
           @fedora
openblas-threads.x86_64                     0.3.6-2.fc30
           @updates
openblas-threads64_.x86_64                  0.3.6-2.fc30
           @updates

ls -l /usr/lib64/R/lib/libRblas.so

-rwxr-xr-x. 1 root root 60204008 Jul  2 10:15 /usr/lib64/R/lib/libRblas.so


lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       46 bits physical, 48 bits virtual
CPU(s):              12
On-line CPU(s) list: 0-11
Thread(s) per core:  2
Core(s) per socket:  6
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               63
Model name:          Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Stepping:            2
CPU MHz:             1197.305
CPU max MHz:         3800.0000
CPU min MHz:         1200.0000
BogoMIPS:            6983.42
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            15360K
NUMA node0 CPU(s):   0-11




B: not working

Installed Packages
blas.x86_64                                 3.8.0-12.fc30
           @updates
liblas.x86_64                               1.8.1-1.fc30
           @fedora
openblas.x86_64                             0.3.6-2.fc30
           @updates
openblas-Rblas.x86_64                       0.3.6-2.fc30
           @updates
openblas-openmp.x86_64                      0.3.6-2.fc30
           @updates
openblas-serial.x86_64                      0.3.6-2.fc30
           @updates
openblas-srpm-macros.noarch                 2-5.fc30
           @fedora
openblas-threads.x86_64                     0.3.6-2.fc30
           @updates
openblas-threads64_.x86_64                  0.3.6-2.fc30
           @updates

-rwxr-xr-x. 1 root root 60204008 Jul  2 10:15 /usr/lib64/R/lib/libRblas.so

lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       46 bits physical, 48 bits virtual
CPU(s):              20
On-line CPU(s) list: 0-19
Thread(s) per core:  2
Core(s) per socket:  10
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz
Stepping:            4
CPU MHz:             1200.180
CPU max MHz:         4500.0000
CPU min MHz:         1200.0000
BogoMIPS:            6600.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            14080K
NUMA node0 CPU(s):   0-19



On Thu, Jul 18, 2019 at 1:05 PM Ivan Krylov <[hidden email]> wrote:

>
> On Thu, 18 Jul 2019 11:50:17 -0400
> Sarah Goslee <[hidden email]> wrote:
>
> > The problem is in the conversion from RGB to Lab.
>
> Hmm. Assuming defaults and skipping all checks, convertColor(red.rgb,
> from = "sRGB", to = "Lab") amounts to the following:
>
> red.rgb <- t(col2rgb(rep('red',8), alpha = 0)/255)
> # let's hope this implementation detail wasn't changed in 3.6.1
> white.point <- grDevices:::c2to3(grDevices:::white.points[,'D65'])
> (red.xyz <- grDevices::colorspaces$sRGB$toXYZ(red.rgb, white.point))[1,]
> # [1] 0.4168213 0.2149235 0.0195385
> (red.lab <- grDevices::colorspaces$Lab$fromXYZ(red.xyz,white.point))[1,] #        L        a        b
> # 53.48418 80.01027 67.38407
> convertColor(red.rgb, from = "sRGB", to = "Lab")[1,]
> #        L        a        b
> # 53.48418 80.01027 67.38407
>
> Does the manual way result in a different output? If so, do the
> differences appear in red.xyz or red.lab?
>
> grDevices::colorspaces$sRGB$toXYZ is pretty simple, just a matrix
> product.  (I hope there are no differences in
> as.list(environment(grDevices::colorspaces$sRGB$toXYZ))[c('dogamma','M')]
> on different computers - that would be strange.)
> grDevices::colorspaces$Lab$fromXYZ looks more complicated but is just a
> bunch of vectorised element-wise algebra, so tracing the intermediate
> results should also be possible.
>
> > And to make it even more bizarre, it fails with eight or more colors,
> > but works correctly with seven or fewer.
>
> This is just a wild guess, but could /usr/lib64/R/lib/libRblas.so per
> chance be a symlink to an OpenBLAS copy? Could any of the computers
> exhibiting the bizarre behaviour be equipped with an AVX-512-capable
> CPU?
>
> --
> Best regards,
> Ivan



--
Sarah Goslee (she/her)
http://www.numberwright.com

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: bizarre color space conversion problem

Ivan Krylov
On Thu, 18 Jul 2019 13:30:09 -0400
Sarah Goslee <[hidden email]> wrote:

> I'm not even remotely a hardware expert: if the difference is due to
> changes in the instruction set, I assume that has potential
> consequences for other things, and I just happened to spot it in this
> particular case because it's visualization-based? (Yikes.)

Yes, this might be bad. I have heard about OpenBLAS (specifically, the
matrix product routine) misbehaving on certain AVX-512 capable
processors, so much that they had to disable some optimizations in
0.3.6 [*], which you already have installed. Still, would `env
OPENBLAS_CORETYPE=Haswell R --vanilla` give a better result?

> As it says in my first email (but way at the bottom), I'd already
> gotten as far as locating the problem in this line from
> grDevices::convertColor()
>
>     xyz <- from$toXYZ(color, from.ref.white)

Thanks for confirming this! It felt that I had to make sure, since the
behaviour we observe is so confusing.

> > (red.xyz <- grDevices::colorspaces$sRGB$toXYZ(red.rgb,
> > white.point))[1,]  
> [1] 0.7733981 0.9280769 0.1383974
> > # [1] 0.4168213 0.2149235 0.0195385

One last check: would

red.rgb %*% as.list(environment(grDevices::colorspaces$sRGB$toXYZ))$M

still produce different results on your computers?

> blas.x86_64                                 3.8.0-12.fc30
> openblas.x86_64                             0.3.6-2.fc30

I do not know enough about Fedora's "alternatives" system, but it does
look like R is using OpenBLAS.

> A: working
> Model name:          Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz

> B: not working
> Model name:          Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz

Again, this points in the direction of OpenBLAS not doing AVX-512 math
properly. Let's hope that OPENBLAS_CORETYPE=Haswell solves it.

--
Best regards,
Ivan

[*]
https://github.com/xianyi/OpenBLAS/issues/1955
https://github.com/xianyi/OpenBLAS/issues/2029
https://github.com/xianyi/OpenBLAS/issues/2168
https://github.com/xianyi/OpenBLAS/issues/2182

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: bizarre color space conversion problem

Sarah Goslee
Wow. You are entirely correct. I would not have been able to pinpoint
the problem, or how to test it. Thank you.

I am unhappy you are right, since these are the fast workstations I
use for all of my heavy-duty analysis, and it's not even *possible* to
rerun everything.

Oh dear.

Here, with the options you suggested, it produces the expected results:

# env OPENBLAS_CORETYPE=Haswell R --vanilla

> white <- c(x = 0.953205712549377, 1, y = 1.08538438164692)
> red10.rgb <- structure(c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), .Dim = c(10L, 3L), .Dimnames = list( NULL, c("r", "g", "b")))
> convertColor(red10.rgb, from = "sRGB", to = "Lab")
             L        a        b
 [1,] 53.48418 80.01027 67.38407
 [2,] 53.48418 80.01027 67.38407
 [3,] 53.48418 80.01027 67.38407
 [4,] 53.48418 80.01027 67.38407
 [5,] 53.48418 80.01027 67.38407
 [6,] 53.48418 80.01027 67.38407
 [7,] 53.48418 80.01027 67.38407
 [8,] 53.48418 80.01027 67.38407
 [9,] 53.48418 80.01027 67.38407
[10,] 53.48418 80.01027 67.38407


> red10.rgb %*% as.list(environment(grDevices::colorspaces$sRGB$toXYZ))$M
           [,1]      [,2]      [,3]
 [1,] 0.4168213 0.2149235 0.0195385
 [2,] 0.4168213 0.2149235 0.0195385
 [3,] 0.4168213 0.2149235 0.0195385
 [4,] 0.4168213 0.2149235 0.0195385
 [5,] 0.4168213 0.2149235 0.0195385
 [6,] 0.4168213 0.2149235 0.0195385
 [7,] 0.4168213 0.2149235 0.0195385
 [8,] 0.4168213 0.2149235 0.0195385
 [9,] 0.4168213 0.2149235 0.0195385
[10,] 0.4168213 0.2149235 0.0195385

On Thu, Jul 18, 2019 at 1:59 PM Ivan Krylov <[hidden email]> wrote:

>
> On Thu, 18 Jul 2019 13:30:09 -0400
> Sarah Goslee <[hidden email]> wrote:
>
> > I'm not even remotely a hardware expert: if the difference is due to
> > changes in the instruction set, I assume that has potential
> > consequences for other things, and I just happened to spot it in this
> > particular case because it's visualization-based? (Yikes.)
>
> Yes, this might be bad. I have heard about OpenBLAS (specifically, the
> matrix product routine) misbehaving on certain AVX-512 capable
> processors, so much that they had to disable some optimizations in
> 0.3.6 [*], which you already have installed. Still, would `env
> OPENBLAS_CORETYPE=Haswell R --vanilla` give a better result?
>
> > As it says in my first email (but way at the bottom), I'd already
> > gotten as far as locating the problem in this line from
> > grDevices::convertColor()
> >
> >     xyz <- from$toXYZ(color, from.ref.white)
>
> Thanks for confirming this! It felt that I had to make sure, since the
> behaviour we observe is so confusing.
>
> > > (red.xyz <- grDevices::colorspaces$sRGB$toXYZ(red.rgb,
> > > white.point))[1,]
> > [1] 0.7733981 0.9280769 0.1383974
> > > # [1] 0.4168213 0.2149235 0.0195385
>
> One last check: would
>
> red.rgb %*% as.list(environment(grDevices::colorspaces$sRGB$toXYZ))$M
>
> still produce different results on your computers?
>
> > blas.x86_64                                 3.8.0-12.fc30
> > openblas.x86_64                             0.3.6-2.fc30
>
> I do not know enough about Fedora's "alternatives" system, but it does
> look like R is using OpenBLAS.
>
> > A: working
> > Model name:          Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
>
> > B: not working
> > Model name:          Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz
>
> Again, this points in the direction of OpenBLAS not doing AVX-512 math
> properly. Let's hope that OPENBLAS_CORETYPE=Haswell solves it.
>
> --
> Best regards,
> Ivan
>
> [*]
> https://github.com/xianyi/OpenBLAS/issues/1955
> https://github.com/xianyi/OpenBLAS/issues/2029
> https://github.com/xianyi/OpenBLAS/issues/2168
> https://github.com/xianyi/OpenBLAS/issues/2182



--
Sarah Goslee (she/her)
http://www.numberwright.com

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: bizarre color space conversion problem

plangfelder
Sarah, if you haven't done so already, please do us (OpenBLAS users) a
big favor and report the bug, either to Fedora or directly to OpenBLAS
maintainers.

Peter

On Thu, Jul 18, 2019 at 11:46 AM Sarah Goslee <[hidden email]> wrote:

>
> Wow. You are entirely correct. I would not have been able to pinpoint
> the problem, or how to test it. Thank you.
>
> I am unhappy you are right, since these are the fast workstations I
> use for all of my heavy-duty analysis, and it's not even *possible* to
> rerun everything.
>
> Oh dear.
>
> Here, with the options you suggested, it produces the expected results:
>
> # env OPENBLAS_CORETYPE=Haswell R --vanilla
>
> > white <- c(x = 0.953205712549377, 1, y = 1.08538438164692)
> > red10.rgb <- structure(c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), .Dim = c(10L, 3L), .Dimnames = list( NULL, c("r", "g", "b")))
> > convertColor(red10.rgb, from = "sRGB", to = "Lab")
>              L        a        b
>  [1,] 53.48418 80.01027 67.38407
>  [2,] 53.48418 80.01027 67.38407
>  [3,] 53.48418 80.01027 67.38407
>  [4,] 53.48418 80.01027 67.38407
>  [5,] 53.48418 80.01027 67.38407
>  [6,] 53.48418 80.01027 67.38407
>  [7,] 53.48418 80.01027 67.38407
>  [8,] 53.48418 80.01027 67.38407
>  [9,] 53.48418 80.01027 67.38407
> [10,] 53.48418 80.01027 67.38407
>
>
> > red10.rgb %*% as.list(environment(grDevices::colorspaces$sRGB$toXYZ))$M
>            [,1]      [,2]      [,3]
>  [1,] 0.4168213 0.2149235 0.0195385
>  [2,] 0.4168213 0.2149235 0.0195385
>  [3,] 0.4168213 0.2149235 0.0195385
>  [4,] 0.4168213 0.2149235 0.0195385
>  [5,] 0.4168213 0.2149235 0.0195385
>  [6,] 0.4168213 0.2149235 0.0195385
>  [7,] 0.4168213 0.2149235 0.0195385
>  [8,] 0.4168213 0.2149235 0.0195385
>  [9,] 0.4168213 0.2149235 0.0195385
> [10,] 0.4168213 0.2149235 0.0195385
>
> On Thu, Jul 18, 2019 at 1:59 PM Ivan Krylov <[hidden email]> wrote:
> >
> > On Thu, 18 Jul 2019 13:30:09 -0400
> > Sarah Goslee <[hidden email]> wrote:
> >
> > > I'm not even remotely a hardware expert: if the difference is due to
> > > changes in the instruction set, I assume that has potential
> > > consequences for other things, and I just happened to spot it in this
> > > particular case because it's visualization-based? (Yikes.)
> >
> > Yes, this might be bad. I have heard about OpenBLAS (specifically, the
> > matrix product routine) misbehaving on certain AVX-512 capable
> > processors, so much that they had to disable some optimizations in
> > 0.3.6 [*], which you already have installed. Still, would `env
> > OPENBLAS_CORETYPE=Haswell R --vanilla` give a better result?
> >
> > > As it says in my first email (but way at the bottom), I'd already
> > > gotten as far as locating the problem in this line from
> > > grDevices::convertColor()
> > >
> > >     xyz <- from$toXYZ(color, from.ref.white)
> >
> > Thanks for confirming this! It felt that I had to make sure, since the
> > behaviour we observe is so confusing.
> >
> > > > (red.xyz <- grDevices::colorspaces$sRGB$toXYZ(red.rgb,
> > > > white.point))[1,]
> > > [1] 0.7733981 0.9280769 0.1383974
> > > > # [1] 0.4168213 0.2149235 0.0195385
> >
> > One last check: would
> >
> > red.rgb %*% as.list(environment(grDevices::colorspaces$sRGB$toXYZ))$M
> >
> > still produce different results on your computers?
> >
> > > blas.x86_64                                 3.8.0-12.fc30
> > > openblas.x86_64                             0.3.6-2.fc30
> >
> > I do not know enough about Fedora's "alternatives" system, but it does
> > look like R is using OpenBLAS.
> >
> > > A: working
> > > Model name:          Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
> >
> > > B: not working
> > > Model name:          Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz
> >
> > Again, this points in the direction of OpenBLAS not doing AVX-512 math
> > properly. Let's hope that OPENBLAS_CORETYPE=Haswell solves it.
> >
> > --
> > Best regards,
> > Ivan
> >
> > [*]
> > https://github.com/xianyi/OpenBLAS/issues/1955
> > https://github.com/xianyi/OpenBLAS/issues/2029
> > https://github.com/xianyi/OpenBLAS/issues/2168
> > https://github.com/xianyi/OpenBLAS/issues/2182
>
>
>
> --
> Sarah Goslee (she/her)
> http://www.numberwright.com
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: bizarre color space conversion problem

Sarah Goslee
Peter, it appears to be the same as this bug:

https://github.com/xianyi/OpenBLAS/issues/2168

I added my info to the discussion.

Thanks for the reminder.

And thank you again to Ivan for the help.

Sarah


On Thu, Jul 18, 2019 at 3:17 PM Peter Langfelder
<[hidden email]> wrote:
>
> Sarah, if you haven't done so already, please do us (OpenBLAS users) a
> big favor and report the bug, either to Fedora or directly to OpenBLAS
> maintainers.
>
> Peter
>

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.