removing outlier

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

removing outlier

Juli
Hey,

i want to remove outliers so I tried do do this:

# 1 define mean and sd
sd.AT_ZU_SPAET <- sd(AT_ZU_SPAET)
mitt.AT_ZU_SPAET <- mean(AT_ZU_SPAET)
#
sd.Anzahl_BAF <- sd(Anzahl_BAF)
mitt.Anzahl_BAF <- mean(Anzahl_BAF)
#
sd.Änderungsintervall <- sd(Änderungsintervall)
mitt.Änderungsintervall <- mean(Änderungsintervall)
#
# 2 identify outliers
DA[ abs(AT_ZU_SPAET - mitt.AT_ZU_SPAET) > ( 3 * sd.AT_ZU_SPAET)  , ]
DA[ abs(Anzahl_BAF - mitt.Anzahl_BAF) > ( 3 * sd.Anzahl_BAF)  , ]
DA[ abs(Änderungsintervall - mitt.Änderungsintervall) > ( 3 * sd.Änderungsintervall)  , ]
#
# 3 remove outliers
AT_ZU_SPAET.clean <- DA[ (abs(AT_ZU_SPAET - mitt.AT_ZU_SPAET) < (3*sd.AT_ZU_SPAET)), ]
Anzahl_BAF.clean <- DA[ (abs(Anzahl_BAF - mitt.Anzahl_BAF) < (3*sd.Anzahl_BAF)), ]
Änderungsintervall.clean <- DA[ (abs(Änderungsintervall - mitt.Änderungsintervall) <
(3*sd.Änderungsintervall)), ]

My problem ist, that I am only able to remove the outliers of one column of my table, but I want to remove the outliers of every column of the table.

Could anybody help me?
Reply | Threaded
Open this post in threaded view
|

Re: removing outlier

Jim Lemon-4
Hi Juli,
What you can do is to make your outlier remover into a function like this:

remove_outlier_by_sd<-function(x,nsd=3) {
 meanx<-mean(x,na.rm=TRUE)
 sdx<-sd(x,na.rm=TRUE)
 return(x[abs(x-xmean) < nsd*sdx])
}

Then apply the function to your data frame ("table")

newDA<-sapply(DA,remove_outlier_by_sd)

newDA will be a list, as it is likely that its elements will be of
different lengths. You may be told that you really shouldn't remove
outliers and learn to love them, but I will leave that to others.

Jim


On Sat, Sep 12, 2015 at 12:15 AM, Juli <[hidden email]> wrote:

> Hey,
>
> i want to remove outliers so I tried do do this:
>
> # 1 define mean and sd
> sd.AT_ZU_SPAET <- sd(AT_ZU_SPAET)
> mitt.AT_ZU_SPAET <- mean(AT_ZU_SPAET)
> #
> sd.Anzahl_BAF <- sd(Anzahl_BAF)
> mitt.Anzahl_BAF <- mean(Anzahl_BAF)
> #
> sd.Änderungsintervall <- sd(Änderungsintervall)
> mitt.Änderungsintervall <- mean(Änderungsintervall)
> #
> # 2 identify outliers
> DA[ abs(AT_ZU_SPAET - mitt.AT_ZU_SPAET) > ( 3 * sd.AT_ZU_SPAET)  , ]
> DA[ abs(Anzahl_BAF - mitt.Anzahl_BAF) > ( 3 * sd.Anzahl_BAF)  , ]
> DA[ abs(Änderungsintervall - mitt.Änderungsintervall) > ( 3 *
> sd.Änderungsintervall)  , ]
> #
> # 3 remove outliers
> AT_ZU_SPAET.clean <- DA[ (abs(AT_ZU_SPAET - mitt.AT_ZU_SPAET) <
> (3*sd.AT_ZU_SPAET)), ]
> Anzahl_BAF.clean <- DA[ (abs(Anzahl_BAF - mitt.Anzahl_BAF) <
> (3*sd.Anzahl_BAF)), ]
> Änderungsintervall.clean <- DA[ (abs(Änderungsintervall -
> mitt.Änderungsintervall) <
> (3*sd.Änderungsintervall)), ]
>
> My problem ist, that I am only able to remove the outliers of one column of
> my table, but I want to remove the outliers of every column of the table.
>
> Could anybody help me?
>
>
>
>
> --
> View this message in context:
> http://r.789695.n4.nabble.com/removing-outlier-tp4712137.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: removing outlier

Juli
Hi Jim,

thank you for your help. :)

My point is, that there are outlier and I don´t really know how to deal with that.

I need the dataframe for a regression and read often that only a few outlier can change your results very much. In addition, regression diacnostics didn´t indcate me the best results.
Yes, and I know its not the core of statistics to work in a way you get results you would like to have ;).

So what is your suggestion?

And if I remove the outliers, my problem ist, that as you said, they differ in length. I need the data frame for a regression, so can I remove the whole column or is there a call to exclude the data?

JULI
Reply | Threaded
Open this post in threaded view
|

Re: removing outlier

David Winsemius

On Sep 12, 2015, at 2:32 AM, Juli wrote:

> Hi Jim,
>
> thank you for your help. :)
>
> My point is, that there are outlier and I don´t really know how to deal with
> that.
>
> I need the dataframe for a regression and read often that only a few outlier
> can change your results very much. In addition, regression diacnostics
> didn´t indcate me the best results.
> Yes, and I know its not the core of statistics to work in a way you get
> results you would like to have ;).
>
> So what is your suggestion?
>
> And if I remove the outliers, my problem ist, that as you said, they differ
> in length. I need the data frame for a regression, so can I remove the whole
> column or is there a call to exclude the data?

Most regression methods have a 'subset' parameter which would allow you to distort the data to your desired specification. But why not think about examining a different statistical model or using robust methods? That way you can keep all your data. (Sounds like you don't really have a lot.)

--
David.

>
> JULI
>
>
>
> --
> View this message in context: http://r.789695.n4.nabble.com/removing-outlier-tp4712137p4712170.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

David Winsemius
Alameda, CA, USA

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: removing outlier

Bert Gunter-2
... and this, of course, is a nice example of how statistics
contributes to the "irreproducibility crisis" now roiling Science.

Cheers,
Bert

(Quote from a long ago engineering colleague: "Whenever I see an
outlier, I never know whether to throw it away or patent it.")


Bert Gunter

"Data is not information. Information is not knowledge. And knowledge
is certainly not wisdom."
   -- Clifford Stoll


On Sat, Sep 12, 2015 at 9:52 AM, David Winsemius <[hidden email]> wrote:

>
> On Sep 12, 2015, at 2:32 AM, Juli wrote:
>
>> Hi Jim,
>>
>> thank you for your help. :)
>>
>> My point is, that there are outlier and I don´t really know how to deal with
>> that.
>>
>> I need the dataframe for a regression and read often that only a few outlier
>> can change your results very much. In addition, regression diacnostics
>> didn´t indcate me the best results.
>> Yes, and I know its not the core of statistics to work in a way you get
>> results you would like to have ;).
>>
>> So what is your suggestion?
>>
>> And if I remove the outliers, my problem ist, that as you said, they differ
>> in length. I need the data frame for a regression, so can I remove the whole
>> column or is there a call to exclude the data?
>
> Most regression methods have a 'subset' parameter which would allow you to distort the data to your desired specification. But why not think about examining a different statistical model or using robust methods? That way you can keep all your data. (Sounds like you don't really have a lot.)
>
> --
> David.
>>
>> JULI
>>
>>
>>
>> --
>> View this message in context: http://r.789695.n4.nabble.com/removing-outlier-tp4712137p4712170.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
> David Winsemius
> Alameda, CA, USA
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: removing outlier

David Winsemius


If this mailing list accepted formatted submissions I would have used the trèsModernSarcastic font for my first sentence. Failing the availability of that mode of communication I am (top) posting through Nabble (perhaps)  in "Comic Sans".
<br />

On Sat, Sep 12, 2015 at 9:52 AM, David Winsemius <[hidden email]> wrote:
>
> On Sep 12, 2015, at 2:32 AM, Juli wrote:

>> And if I remove the outliers, my problem ist, that as you said, they differ
>> in length. I need the data frame for a regression, so can I remove the whole
>> column or is there a call to exclude the data?
>
> Most regression methods have a 'subset' parameter which would allow you to distort the data to your desired specification.

Bert Gunter-2 wrote
... and this, of course, is a nice example of how statistics
contributes to the "irreproducibility crisis" now roiling Science.


Cheers,
Bert

(Quote from a long ago engineering colleague: "Whenever I see an
outlier, I never know whether to throw it away or patent it.")


Bert Gunter

"Data is not information. Information is not knowledge. And knowledge
is certainly not wisdom."
   -- Clifford Stoll


On Sat, Sep 12, 2015 at 9:52 AM, David Winsemius <[hidden email]> wrote:
>
> On Sep 12, 2015, at 2:32 AM, Juli wrote:
>
>> Hi Jim,
>>
>> thank you for your help. :)
>>
>> My point is, that there are outlier and I don´t really know how to deal with
>> that.
>>
>> I need the dataframe for a regression and read often that only a few outlier
>> can change your results very much. In addition, regression diacnostics
>> didn´t indcate me the best results.
>> Yes, and I know its not the core of statistics to work in a way you get
>> results you would like to have ;).
>>
>> So what is your suggestion?
>>
>> And if I remove the outliers, my problem ist, that as you said, they differ
>> in length. I need the data frame for a regression, so can I remove the whole
>> column or is there a call to exclude the data?
>
> Most regression methods have a 'subset' parameter which would allow you to distort the data to your desired specification. But why not think about examining a different statistical model or using robust methods? That way you can keep all your data. (Sounds like you don't really have a lot.)
>
> --
> David.
>>
>> JULI
>>
>>
>>
>> --
>> View this message in context: http://r.789695.n4.nabble.com/removing-outlier-tp4712137p4712170.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
> David Winsemius
> Alameda, CA, USA
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: removing outlier --> use robust regression !

Martin Maechler
In reply to this post by Juli
>>>>> Juli  <[hidden email]>
>>>>>     on Sat, 12 Sep 2015 02:32:39 -0700 writes:

     > Hi Jim, thank you for your help. :)

     > My point is, that there are outlier and I don´t really
     > know how to deal with that.

     > I need the dataframe for a regression and read often that
     > only a few outlier can change your results very much. In
     > addition, regression diacnostics didn´t indcate me the
     > best results.  Yes, and I know its not the core of
     > statistics to work in a way you get results you would
     > like to have ;).

     > So what is your suggestion?

Use robust regression, e.g.
    MASS::rlm()  {part of every R installation},
or a somewhat better and more sophisticated version.
    lmrob()  from package 'robustbase' {yes, shameless promotion}.

Further:

1) Removing outliers is not at all the best way to deal with such
  problems (intuitively, because it is a *dis*continuous method).
  Rather they should be downweighted (continuously, as it
  happens with methods used in  rlm() or lmrob() see above)

2) Removing outliers in *multivariate* setting, if you want to do
   it in spite of 1)  by using univariate treatment {each column
   separately as you do here} is often completely insufficient.  E.g.
   the bivariate outlier  in
      xy <- cbind(x= c(2,1:9), y=c(8,1:9));  plot(xy)
   cannot be found by looking at 'x' and 'y' separately.
   
3) If, in spite of 1) and 2) you are considering univariate
   treatment, using mean() and sd() for detecting univariate outliers
   has been proven to be insufficient more than 50 years ago (*1), and
   if one looks closer into the literature (say "L_1") even
   considerably longer ago.
   Using  median() and mad() instead, is one possibility (*2) of
   what you should do. Hampel's rule (*3)
   proposes declaring outliers for the observations outside
   the interval   median(x) +/- 3.5*mad(x)


*1 Tukey, J. W. (1960) A survey of sampling from contaminated distributions.
   In Contributions to Probability and Statistics,
   eds I. Olkin, S. Ghurye, W. Hoeffding, W. Madow and H. Mann,
   pp. 448–485. Stanford: Stanford University Press.

*2 Another (less robust, but still infinitely better than mean/sd) approach
   uses  median() and IQR() which is
   basically/approximately what boxplots do to identify outliers.


*3 Frank R. Hampel (1985)
   The Breakdown Points of the Mean Combined With Some Rejection Rules,
   Technometrics, 27:2, 95-107
          [ http://dx.doi.org/10.1080/00401706.1985.10488027 ]

   See also section
       "1.4b. How Well Are Objective and Subjective Metbods for
       the ReJection of Outliers Doing in the Context of Robust
       Estimation?",
    page 62 ff  od
  of
    Frank R. Hampel, Elvezio M. Ronchetti, Peter J. Rousseeuw and Werner A. Stahel
    (1986) Robust Statistics: The Approach Based on Influence Functions.
    John Wiley & Sons, Inc.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.