Quantcast

Using data.table code in an interactive desktop application written in C# or Java

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Using data.table code in an interactive desktop application written in C# or Java

Don Brady
I have written a largish R data.table application for back-testing option trading strategies.   It involves large quantities of option price data, all of which is stored in data.tables.  As an R application, it works fast and well, thanks to the strengths of data.table.  There are about 10,000 lines of R code, almost all of which are calling data.table functions.

However, I want to take some of the code and create a snappy desktop GUI application for quick interactive option portfolio and trade evaluation.  Ordinarily I would do this in Java or C#.  However, since the prior code is all written using data.table calls, it would be ideal if I could continue to use.table.

I tried Shiny and I  got a shiny app. written and working, that includes lots of use of data.table and my existing tables.  However, it is too slow.  I need lightning fast for the GUI which is to be exploratory.

So, after this background, my question is, is there any way to call data.table from a C# or Java desktop GUI app. without involving R?  Assuming the answer is No, what would be another in-memory database implementation that can be called from C# or Java, that would be similar enough to data.table that I would be able to migrate over my data.table code over.   It is obviously no problem to convert over the data, since that is just columns.  But my concern is to minimize the effort in converting over the code, which is now, of course, full of data-table joins, merges, selects, renames, updates etc., all currently written in data.table's very unique syntax (currently coded in R function calls).  Is there some in-memory database package that would support a similar syntax to facilitate converting the code from data.table?

Thank you very much, I should add, for the wonderful data.table package!  I certainly will not abandon it.  I am just looking for something somewhat compatible with its style, for a limited special purpose.

Don



_______________________________________________
datatable-help mailing list
[hidden email]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using data.table code in an interactive desktop application written in C# or Java

Santosh Srinivas
Hi Don,

I'm not sure if this will meet your performance expectations from the GUI.

But, I am creating my GUI using HTML 5 stuff on Electron (https://github.com/electron/electron) which fires off calls to my  OpenCPU (https://www.opencpu.org/) service that does most of backend R related work.

I'm guessing that you could run Java applets using a similar setup.

Thank you.
Santosh


On Sat, Apr 15, 2017 at 8:49 AM, Don Brady <[hidden email]> wrote:
I have written a largish R data.table application for back-testing option trading strategies.   It involves large quantities of option price data, all of which is stored in data.tables.  As an R application, it works fast and well, thanks to the strengths of data.table.  There are about 10,000 lines of R code, almost all of which are calling data.table functions.

However, I want to take some of the code and create a snappy desktop GUI application for quick interactive option portfolio and trade evaluation.  Ordinarily I would do this in Java or C#.  However, since the prior code is all written using data.table calls, it would be ideal if I could continue to use.table.

I tried Shiny and I  got a shiny app. written and working, that includes lots of use of data.table and my existing tables.  However, it is too slow.  I need lightning fast for the GUI which is to be exploratory.

So, after this background, my question is, is there any way to call data.table from a C# or Java desktop GUI app. without involving R?  Assuming the answer is No, what would be another in-memory database implementation that can be called from C# or Java, that would be similar enough to data.table that I would be able to migrate over my data.table code over.   It is obviously no problem to convert over the data, since that is just columns.  But my concern is to minimize the effort in converting over the code, which is now, of course, full of data-table joins, merges, selects, renames, updates etc., all currently written in data.table's very unique syntax (currently coded in R function calls).  Is there some in-memory database package that would support a similar syntax to facilitate converting the code from data.table?

Thank you very much, I should add, for the wonderful data.table package!  I certainly will not abandon it.  I am just looking for something somewhat compatible with its style, for a limited special purpose.

Don



_______________________________________________
datatable-help mailing list
[hidden email]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help


_______________________________________________
datatable-help mailing list
[hidden email]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using data.table code in an interactive desktop application written in C# or Java

mbacou
Don,
In addition to Santosh's suggestion below, you could also read about
these 2:

- Rserve, a TCP/IP server that's often used to connect applications to
local R processes (e.g. Tableau uses Rserve). I've had good results with
Rserve in the past in a use-case similar to yours. I would just open a
socket to a running Rserve session preloaded with my custom R packages
and return only slices of data back to a web GUI.

- rJava (an R interface for Java) used for example in the StatET
Java-based IDE for R

Both options would allow you to keep your core data and business logic
in your current R package.

--Mel.

On 04/15/2017 12:28 AM, Santosh Srinivas wrote:

> Hi Don,
>
> I'm not sure if this will meet your performance expectations from the GUI.
>
> But, I am creating my GUI using HTML 5 stuff on Electron
> (https://github.com/electron/electron) which fires off calls to my
>  OpenCPU (https://www.opencpu.org/) service that does most of backend
> R related work.
>
> I'm guessing that you could run Java applets using a similar setup.
>
> Thank you.
> Santosh
>
>
> On Sat, Apr 15, 2017 at 8:49 AM, Don Brady <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I have written a largish R data.table application for back-testing
>     option trading strategies.   It involves large quantities of
>     option price data, all of which is stored in data.tables. As an R
>     application, it works fast and well, thanks to the strengths of
>     data.table.  There are about 10,000 lines of R code, almost all of
>     which are calling data.table functions.
>
>     However, I want to take some of the code and create a snappy
>     desktop GUI application for quick interactive option portfolio and
>     trade evaluation.  Ordinarily I would do this in Java or C#.
>     However, since the prior code is all written using data.table
>     calls, it would be ideal if I could continue to use.table.
>
>     I tried Shiny and I  got a shiny app. written and working, that
>     includes lots of use of data.table and my existing tables.
>     However, it is too slow.  I need lightning fast for the GUI which
>     is to be exploratory.
>
>     So, after this background, my question is, is there any way to
>     call data.table from a C# or Java desktop GUI app. without
>     involving R?  Assuming the answer is No, what would be another
>     in-memory database implementation that can be called from C# or
>     Java, that would be similar enough to data.table that I would be
>     able to migrate over my data.table code over.   It is obviously no
>     problem to convert over the data, since that is just columns.  But
>     my concern is to minimize the effort in converting over the code,
>     which is now, of course, full of data-table joins, merges,
>     selects, renames, updates etc., all currently written in
>     data.table's very unique syntax (currently coded in R function
>     calls).  Is there some in-memory database package that would
>     support a similar syntax to facilitate converting the code from
>     data.table?
>
>     Thank you very much, I should add, for the wonderful data.table
>     package!  I certainly will not abandon it.  I am just looking for
>     something somewhat compatible with its style, for a limited
>     special purpose.
>
>     Don
>
>
>
>     _______________________________________________
>     datatable-help mailing list
>     [hidden email]
>     <mailto:[hidden email]>
>     https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
>     <https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help>
>
>
>
>
> _______________________________________________
> datatable-help mailing list
> [hidden email]
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help

_______________________________________________
datatable-help mailing list
[hidden email]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using data.table code in an interactive desktop application written in C# or Java

Don Brady
Thanks you very much for the comments.  Those are some good ways to get he data back and forth.  In fact, I think we can regard those mechanical issues as taken care of.  I can see a number of ways to do that.

However, once I get the columnar data (what was data.tables in R) into C# and stored in memory, what representation of it should I use in C# that would be most like data.table?   And how would I manipulate it then and there in C#, but in data.table like ways, without involving R at that point.  Does anyone know what .Net way would be most like data.table?  Would it be LINQ Objects, for example, or some in-memory data package?   I do not want to do SQL etc. which is so painful compared to data.table.  I know this is not a data.table question as such, but people here are the only people who can understand it because they know what data.table is.  I want to be able to add one column to another etc. as I can do in data.table.  (Then maybe eventually send it back to R as a data.table which is just mechanics again).

If I am too far off topic let me know and I apologize.

Thanks,

Don

_______________________________________________
datatable-help mailing list
[hidden email]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using data.table code in an interactive desktop application written in C# or Java

natus
Hi,

When manipulating huge datasets, the bottleneck is certainly not the
viz tool (shiny, c#, html5 etc).
This won't be data-manipulation part (you won't find faster than
data.table with c# manipulation since most part of data.table is C based).
The slow part of your tool is probably the data-loading phase.

Open-CPU allows to cache data when the opencpu services is starting.
This means each user querying your dataset won't have to wait until the
dataset is loaded into open-cpu memory. see
http://stackoverflow.com/questions/12468574/are-data-objects-persistent-in-opencpus-r-session
Rserver and other stuff are quite old comparing to open-cpu. And I would
go with them.

Keep separate your view (html5, c#), backend (opencpu/R) and database
(opencpu/data.table). And one day you can moove to more flexible
architecture such backend (java/c#) and  database (distributed such hive) if
your dataset cannot feet into RAM memory.

But the opencpu way allows you to fast deploy something compatible with
your 10K lines of data.table/R code

Le 16 avril 2017 à 13:22, Don Brady écrivait :

> Thanks you very much for the comments.  Those are some good ways to get he data
> back and forth.  In fact, I think we can regard those mechanical issues as
> taken care of.  I can see a number of ways to do that.
>
> However, once I get the columnar data (what was data.tables in R) into C# and
> stored in memory, what representation of it should I use in C# that would be
> most like data.table?   And how would I manipulate it then and there in C#, but
> in data.table like ways, without involving R at that point.  Does anyone know
> what .Net way would be most like data.table?  Would it be LINQ Objects, for
> example, or some in-memory data package?   I do not want to do SQL etc. which
> is so painful compared to data.table.  I know this is not a data.table question
> as such, but people here are the only people who can understand it because they
> know what data.table is.  I want to be able to add one column to another etc.
> as I can do in data.table.  (Then maybe eventually send it back to R as a
> data.table which is just mechanics again).
>
> If I am too far off topic let me know and I apologize.
>
> Thanks,
>
> Don

> _______________________________________________
> datatable-help mailing list
> [hidden email]
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help

_______________________________________________
datatable-help mailing list
[hidden email]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using data.table code in an interactive desktop application written in C# or Java

Don Brady
In reply to this post by mbacou
Thanks for the comment involving Open-CPU which is important.  Definitely I will at least still base the data.tables and a lot of the code permanently in R's data table..  I have no doubt that R's data.table is far faster than anything else.  The problem is in the lack of a high-speed non-web based user interface for a richdesktop app. (other than using a client-server architecture).

For highly interactive use, I found Shiny and rhandsontable too sluggish for huge tables.

Looking for what is the nearest thing to R's data.table on the .Net platform itself, for those cases where that is needed, F# and Deedle seem to be very attractive possibilities that I am now exploring.

http://bluemountaincapital.github.io/Deedle/csharpframe.html
http://dacrook.com/battle-of-the-programming-languages/

Deedle is an easy to use library for data and time series manipulation and for scientific programming. It supports working with structured data frames, ordered and unordered data, as well as time series. Deedle is designed to work well for exploratory programming using F# and C# interactive console, but can be also used in efficient compiled .NET code.

The library implements a wide range of operations for data manipulation including advanced indexing and slicing, joining and aligning data, handling of missing values, grouping and aggregation, statistics and more.

The above feature set sounds very similar to R's data.table! 

I am sure that data.table is far superior in performance and function to Deedle  (partly based on this article: http://www.extremeoptimization.com/Blog/index.php/2014/01/data-frame-library-preview/ ).  However, Deedle may be "fast enough" in the subset of cases where I do need to work with directly with data frames  in the front-end C# desktop app for interactive speed.





_______________________________________________
datatable-help mailing list
[hidden email]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
Loading...