Use logical expresion on ff object

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Use logical expresion on ff object

Krystian Radlak
I want to do operation with ff vector from packages ff like simple operation in R like this. (I have read data from file)
Example
a=1:10
b=a[a>5]

how to do this operation with ff packages:
Example :
a=ff(1, length=15*10^7)

I know that I could get logical vector like this:

k=bit(15*10^7) # from bit packages
a=ff(-1,15*10^7)
system.time(
                for (i in chunk(a)){ k[i]<- a[i]<1}
)

Any suggestions?

Best,
Krystian

Reply | Threaded
Open this post in threaded view
|

Re: Use logical expresion on ff object

Łukasz Ręcławowicz
2011/8/18 Krystian Radlak <[hidden email]>

> Any suggestions?
>

Do you really, really need that loop and chunk...?!


--
Mi³ego dnia

        [[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Use logical expresion on ff object

Krystian Radlak
Yes, it's necessery. Operator < is not definied in ff packages.

str(a<1)
 logi(0)

Mayby it's possible to do the same thing easier...
Reply | Threaded
Open this post in threaded view
|

Re: Use logical expresion on ff object

Łukasz Ręcławowicz
2011/8/18 Krystian Radlak <[hidden email]>

> Yes, it's necessery
>

Really? How about something like this:

system.time(k[i]<-a[i]<1)

It should be ok.

--
Mi³ego dnia

        [[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Use logical expresion on ff object

Łukasz Ręcławowicz
W dniu 18 sierpnia 2011 16:26 u¿ytkownik £ukasz Rêc³awowicz <
[hidden email]> napisa³:

>
> It should be ok.
>

It wasn't, sorry! This seems to do right job.

library(ff)
a=ff(1, length=15*10^2)
k=bit(15*10^2)
a=ff(-1,15*10^2)
system.time(k<-a[,i]<1)


--
Mi³ego dnia

        [[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Use logical expresion on ff object

Łukasz Ręcławowicz
In reply to this post by Krystian Radlak
2011/8/18 Krystian Radlak <[hidden email]>

>
> Mayby it's possible to do the same thing easier...
>
> You shouldn't be disappointed ;-)

> a=ff(1, length=15*10^6)
> k=bit(15*10^6)
> a=ff(2,15*10^6)
> system.time(k<-a[,chunk(1,length(a),1e4)]<1)
user system elapsed
0.40 0.23 0.72

--
Mi³ego dnia

        [[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Use logical expresion on ff object

Krystian Radlak
It's not corect. Vector with length 15*10^6 you could store in memory without ff packages.
For vector 15*10^7

 k=bit(15*10^7)
 a=ff(2,15*10^7)
 system.time(k<-a[,chunk(1,length.ff(a),1e4)]<1)

Error: cannot allocate vector of size 572.2 Mb
In addition: Warning messages:
1: In system.time(k <- a[, chunk(1, length.ff(a), 10000)] < 1) :
  Reached total allocation of 1535Mb: see help(memory.size)
2: In system.time(k <- a[, chunk(1, length.ff(a), 10000)] < 1) :
  Reached total allocation of 1535Mb: see help(memory.size)
3: In system.time(k <- a[, chunk(1, length.ff(a), 10000)] < 1) :
  Reached total allocation of 1535Mb: see help(memory.size)
4: In system.time(k <- a[, chunk(1, length.ff(a), 10000)] < 1) :
  Reached total allocation of 1535Mb: see help(memory.size)
Timing stopped at: 2.13 0.84 10.45

And you don't understand my question. In simple operation in R I could do:
a=c(1,4,5,6)
b=a[a>1]
b
[1] 4 5 6

If a is ff object (for example length=15*10^7) this operation doesn't work. Mayby I don't understand something or this operation is not definied currently in this package.
In summarizing:
I have very long vector stored in ff object and I want to chse elements from this vector which fulfil logical expresion (for example a>1).
Reply | Threaded
Open this post in threaded view
|

Re: Use logical expresion on ff object

Łukasz Ręcławowicz
2011/8/19 Krystian Radlak <[hidden email]>

> I have very long vector stored in ff object and I want to chse elements
> from
>  this vector which fulfil logical expresion (for example a>1)
>

?ffvecapply
k<-ffvecapply(a[i1:i2]>1, X=a, BATCHSIZE=5000,RETURN=T)





--
Mi³ego dnia

        [[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel