>>>>> "RobCar" == Carnell, Rob C <

[hidden email]>

>>>>> on Sun, 30 Jul 2006 19:42:29 -0400 writes:

RobCar> NIST maintains a repository of Statistical Reference

RobCar> Datasets at

http://www.itl.nist.gov/div898/strd/. I

RobCar> have been working through the datasets to compare

RobCar> R's results to their references with the hope that

RobCar> if all works well, this could become a validation

RobCar> package.

RobCar> All the linear regression datasets give results with

RobCar> some degree of accuracy except one. The NIST model

RobCar> includes 11 parameters, but R will not compute the

RobCar> estimates for all 11 parameters because it finds the

RobCar> data matrix to be singular.

RobCar> The code I used is below. Any help in getting R to

RobCar> estimate all 11 regression parameters would be

RobCar> greatly appreciated.

RobCar> I am posting this to the R-devel list since I think

RobCar> that the discussion might involve the limitations of

RobCar> platform precision.

RobCar> I am using R 2.3.1 for Windows XP.

RobCar> rm(list=ls())

RobCar> require(gsubfn)

RobCar> defaultPath <- "my path"

RobCar> data.base <- "

http://www.itl.nist.gov/div898/strd/lls/data/LINKS/DATA"

Here is a slight improvement {note the function file.path(); and

model <- ..; also poly(V2, 10) !}

which shows you how to tell lm() to "believe" in 10 digit

precision of input data.

-------------------------------------------------------------------------------

reg.data <- paste(data.base, "/Filip.dat", sep="")

filePath <- file.path(defaultPath, "NISTtest.dat")

download.file(reg.data, filePath, quiet=TRUE)

A <- read.table(filePath, skip=60, strip.white=TRUE)

## If you really need high-order polynomial regression in S and R,

## *DO* as you are told in all good books, and use orthogonal polynomials:

(lm.ok <- lm(V1 ~ poly(V2,10), data = A))

## and there is no problem

summary(lm.ok)

## But if you insist on doing nonsense ....

model <- "V1 ~ V2+ I(V2^2)+I(V2^3)+I(V2^4)+I(V2^5)+I(V2^6)+I(V2^7)+I(V2^8)+I(V2^9)+I(V2^10)"

## MM: "better":

(model <- paste("V1 ~ V2", paste("+ I(V2^", 2:10, ")", sep='', collapse='')))

(form <- formula(model))

mod.mat <- model.matrix(form, data = A)

dim(mod.mat) ## 82 11

(m.qr <- qr(mod.mat ))$rank # -> 10 (only, instead of 11)

(m.qr <- qr(mod.mat, tol = 1e-10))$rank # -> 11

(lm.def <- lm(form, data = A)) ## last coef. is NA

(lm.plus <- lm(form, data = A, tol = 1e-10))## no NA coefficients

-------------------------------------------------------------------------------

RobCar> reg.data <- paste(data.base, "/Filip.dat", sep="")

RobCar> model <-

RobCar> "V1~V2+I(V2^2)+I(V2^3)+I(V2^4)+I(V2^5)+I(V2^6)+I(V2^7)+I(V2^8)+I(V2^9)+I

RobCar> (V2^10)"

RobCar> filePath <- paste(defaultPath, "//NISTtest.dat", sep="")

RobCar> download.file(reg.data, filePath, quiet=TRUE)

RobCar> A <- read.table(filePath, skip=60, strip.white=TRUE)

RobCar> lm.data <- lm(formula(model), A)

RobCar> lm.data

RobCar> Rob Carnell

A propos NIST StRD:

If you go further to NONlinear regression,

and use nls(), you will see that high quality statistics

packages such as R do *NOT* always conform to NIST -- at least

not to what NIST did about 5 years ago when I last looked.

There are many nonlinear least squares problems where the

correct result is *NO CONVERGENCE* (because of

over-parametrization, ill-posednes, ...),

owever many (cr.p) pieces of software do "converge"---falsely.

I think you find more on this topic in the monograph of

Bates and Watts (1988), but in any case,

just install and use the CRAN R package 'NISTnls' by Doug Bates

which contains the data sets with documentation and example

calls.

Martin Maechler, ETH Zurich

______________________________________________

[hidden email] mailing list

https://stat.ethz.ch/mailman/listinfo/r-devel