Creating a private CRAN with webpages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Creating a private CRAN with webpages

Joshua Bradley
Hello,

I am trying to setup a private CRAN for work (behind a firewall). The best
options available include miniCRAN
<https://github.com/RevolutionAnalytics/miniCRAN>, drat
<https://github.com/eddelbuettel/drat> and packrat
<https://rstudio.github.io/packrat/>. One problem is these packages do not
automatically generate the web pages that are on the CRAN.

Examples:
https://cran.r-project.org/web/packages/index.html
https://cran.r-project.org/web/packages/available_packages_by_name.html

Each time the CRAN adds a package, there must be an automated process in
place to regenerate the pages again with the new package added (example - A3
<https://cran.r-project.org/web/packages/A3/index.html>). I read somewhere
(possibly on stackoverflow) that the CRAN html pages are statically built.
I would like for users to be able to explore the packages in my private
CRAN just like the public CRAN without having to open R and search for
packages/documentation through the command line.

The R Manual includes a small section
<https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Setting-up-a-package-repository>
on setting up a repository but it only discusses the structure of the
directories needed to host packages. Nothing is mentioned about how the
CRAN creates/updates the /web directory. What is the best way to
generate/maintain the web pages for a private CRAN?

P.S. Let me know if this question is best answered on one of the other
mailing lists.

Josh Bradley

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Creating a private CRAN with webpages

Rainer M Krug-3
If I understand you correctly, you want to have a mirror of CRAN on a private server behind your firewall. Check out https://cran.rstudio.com/mirror-howto.html <https://cran.rstudio.com/mirror-howto.html> which gives instructions on how to do this.

Cheers,

Rainer

> On 8 Jun 2017, at 23:29, Joshua Bradley <[hidden email]> wrote:
>
> Hello,
>
> I am trying to setup a private CRAN for work (behind a firewall). The best
> options available include miniCRAN
> <https://github.com/RevolutionAnalytics/miniCRAN>, drat
> <https://github.com/eddelbuettel/drat> and packrat
> <https://rstudio.github.io/packrat/>. One problem is these packages do not
> automatically generate the web pages that are on the CRAN.
>
> Examples:
> https://cran.r-project.org/web/packages/index.html
> https://cran.r-project.org/web/packages/available_packages_by_name.html
>
> Each time the CRAN adds a package, there must be an automated process in
> place to regenerate the pages again with the new package added (example - A3
> <https://cran.r-project.org/web/packages/A3/index.html>). I read somewhere
> (possibly on stackoverflow) that the CRAN html pages are statically built.
> I would like for users to be able to explore the packages in my private
> CRAN just like the public CRAN without having to open R and search for
> packages/documentation through the command line.
>
> The R Manual includes a small section
> <https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Setting-up-a-package-repository>
> on setting up a repository but it only discusses the structure of the
> directories needed to host packages. Nothing is mentioned about how the
> CRAN creates/updates the /web directory. What is the best way to
> generate/maintain the web pages for a private CRAN?
>
> P.S. Let me know if this question is best answered on one of the other
> mailing lists.
>
> Josh Bradley
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

University of Zürich

Cell:       +41 (0)78 630 66 57

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      [hidden email]

Skype:      RMkrug


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

signature.asc (465 bytes) Download Attachment
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Creating a private CRAN with webpages

Joshua Bradley
In reply to this post by Joshua Bradley
I'm not trying to create a mirror of the CRAN. I also do not want to create
a "subset" of CRAN packages. Sorry if my explanation was confusing. There
are internal proprietary R packages that I would like to host on a private
repo. I like the CRAN web pages that allow a user to browse the various
packages from a browser without having to first install them. I want to
duplicate the process that CRAN goes through when it generate the webpages.

Josh Bradley

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Creating a private CRAN with webpages

Martin Maechler
In reply to this post by Rainer M Krug-3
>>>>> Rainer Krug <[hidden email]>
>>>>>     on Fri, 9 Jun 2017 08:36:31 +0200 writes:

    > If I understand you correctly, you want to have a mirror
    > of CRAN on a private server behind your firewall. Check
    > out https://cran.rstudio.com/mirror-howto.html
    > <https://cran.rstudio.com/mirror-howto.html> which gives
    > instructions on how to do this.  Cheers,

    > Rainer

Yes, definitely:  Just use your private CRAN mirror... and instead
of the above URL, one with the same speed (and AFAIK still
sponsored by the same monney) is
  https://cloud.r-project.org/mirror-howto.html

There it tells you to use  rsync, and I strongly recommend you do!  

Martin Maechler,
ETH Zurich
(running one of the oldest - public - CRAN Mirrors)


    >> On 8 Jun 2017, at 23:29, Joshua Bradley
    >> <[hidden email]> wrote:
    >>
    >> Hello,
    >>
    >> I am trying to setup a private CRAN for work (behind a
    >> firewall). The best options available include miniCRAN
    >> <https://github.com/RevolutionAnalytics/miniCRAN>, drat
    >> <https://github.com/eddelbuettel/drat> and packrat
    >> <https://rstudio.github.io/packrat/>. One problem is
    >> these packages do not automatically generate the web
    >> pages that are on the CRAN.
    >>
    >> Examples:
    >> https://cran.r-project.org/web/packages/index.html
    >> https://cran.r-project.org/web/packages/available_packages_by_name.html
    >>
    >> Each time the CRAN adds a package, there must be an
    >> automated process in place to regenerate the pages again
    >> with the new package added (example - A3
    >> <https://cran.r-project.org/web/packages/A3/index.html>). I
    >> read somewhere (possibly on stackoverflow) that the CRAN
    >> html pages are statically built.  I would like for users
    >> to be able to explore the packages in my private CRAN
    >> just like the public CRAN without having to open R and
    >> search for packages/documentation through the command
    >> line.
    >>
    >> The R Manual includes a small section
    >> <https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Setting-up-a-package-repository>
    >> on setting up a repository but it only discusses the
    >> structure of the directories needed to host
    >> packages. Nothing is mentioned about how the CRAN
    >> creates/updates the /web directory. What is the best way
    >> to generate/maintain the web pages for a private CRAN?
    >>
    >> P.S. Let me know if this question is best answered on one
    >> of the other mailing lists.
    >>
    >> Josh Bradley
    >>
    >> [[alternative HTML version deleted]]
    >>
    >> ______________________________________________
    >> [hidden email] mailing list
    >> https://stat.ethz.ch/mailman/listinfo/r-devel

    > --
    > Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc
    > (Conservation Biology, UCT), Dipl. Phys. (Germany)

    > University of Zürich

    > Cell: +41 (0)78 630 66 57

    > Fax (D): +49 - (0)3 21 21 25 22 44

    > email: [hidden email]

    > Skype: RMkrug

    > xapplication/pgp-signature [Click mouse-2 to save to a
    > file]

    > ----------------------------------------------------------------------
    > ______________________________________________
    > [hidden email] mailing list
    > https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Creating a private CRAN with webpages

Dirk Eddelbuettel
In reply to this post by Joshua Bradley

On 9 June 2017 at 02:44, Joshua Bradley wrote:
| repo. I like the CRAN web pages that allow a user to browse the various
| packages from a browser without having to first install them. I want to
| duplicate the process that CRAN goes through when it generate the webpages.

As your initial email correctly identified, none of miniCRAN, drat, ... do
this currently.

So this is how open source works: you identified a need, now you can go off
and fill it. ;-) You have the PACKAGES{,.gz,.rds} files from your repo. So
accessing packageDescription() for each package is a given.  I do so in
CRANberries for new packages, and while that (ugly) code is (still) not
public, the relevant function for that is

writeNewBlogEntry <- function(curPkg, curVer, reposurl) {
    ## and compute some sort of output for new package -- maybe get description ? NB does NOT work for BioC
    blogpost <- file.path(blogInputDir, "new", paste(curPkg, "_", curVer, ".txt", sep=""))
    con <- file(blogpost, "wt")
    cat("New package", curPkg, "with initial version", curVer,"\n\n", file=con)
    dcf <- read.dcf( url(getDescriptionUrl(curPkg, reposurl)) )
    for (i in 1:ncol(dcf)) {
        cat("<strong>", colnames(dcf)[i], "</strong>: ", htmlEscape(dcf[1,i]), "<br>\n", sep="", file=con)
    }
    closeBlogPost(con, reposurl, curPkg)
}

(The read.dcf() off the package URL is no longer needed with CRAN_package_db().)

A corresponding file for the most recent 'new' package:

-----------------------------------------------------------------------------
New package Risk with initial version 1.0

<strong>Package</strong>: Risk<br>
<strong>Type</strong>: Package<br>
<strong>Title</strong>: Computes 26 Financial Risk Measures for Any Continuous
Distribution<br>
<strong>Version</strong>: 1.0<br>
<strong>Date</strong>: 2017-06-05<br>
<strong>Author</strong>: Saralees Nadarajah, Stephen Chan<br>
<strong>Maintainer</strong>: Saralees Nadarajah &lt;[hidden email]&gt;<br>
<strong>Depends</strong>: R (&gt;= 3.0.1)<br>
<strong>Description</strong>: Computes 26 financial risk measures for any continuous distribution.  The 26 financial risk measures  include value at risk, expected shortfall due to Artzner et al. (1999) &lt;DOI:10.1007/s10957-011-9968-2&gt;, tail conditional median due to Kou et al. (2013) &lt;DOI:10.1287/moor.1120.0577&gt;, expectiles due to Newey and Powell (1987) &lt;DOI:10.2307/1911031&gt;, beyond value at risk due to Longin (2001) &lt;DOI:10.3905/jod.2001.319161&gt;, expected proportional shortfall due to Belzunce et al. (2012) &lt;DOI:10.1016/j.insmatheco.2012.05.003&gt;, elementary risk measure due to Ahmadi-Javid (2012) &lt;DOI:10.1007/s10957-011-9968-2&gt;, omega due to Shadwick and Keating (2002), sortino ratio due to Rollinger and Hoffman (2013), kappa  due to Kaplan and Knowles  (2004), Wang (1998)'s &lt;DOI:10.1080/10920277.1998.10595708&gt; risk measures, Stone (1973)'s &lt;DOI:10.2307/2978638&gt; risk measures, Luce (1980)'s &lt;DOI:10.1007/BF00135033&gt; risk measures
 , Sarin (1987)'s &lt;DOI:10.1007/BF00126387&gt; risk measures, Bronshtein and Kurelenkova (2009)'s risk measures.<br>
<strong>License</strong>: GPL (&gt;= 2)<br>
<strong>Packaged</strong>: 2017-06-08 13:57:28 UTC; mbbsssn2<br>
<strong>NeedsCompilation</strong>: no<br>
<strong>Repository</strong>: CRAN<br>
<strong>Date/Publication</strong>: 2017-06-08 15:19:54 UTC<br>

<p>
<a href="https://cran.r-project.org/package=Risk">More information about Risk at CRAN</a>
-----------------------------------------------------------------------------

If I did it today, I'd start with markdown, but you get the idea.  The page
is the (current) top of http://dirk.eddelbuettel.com/cranberries/cran/new/

I write these as minimal html (as I wrote that code ~ 10 years ago in the
pre-pandoc days) which then get fed into the blog engine.  That helps keeping
content and presentation (css, ...) separate.

I think you have everything you need in the PACKAGES.rds which you get back
as a (large) data.frame from tools::CRAN_package_db().

Hope this helps,  Dirk



--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]
______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Creating a private CRAN with webpages

Prof Brian Ripley
In reply to this post by Joshua Bradley
On 09/06/2017 07:44, Joshua Bradley wrote:
> I'm not trying to create a mirror of the CRAN. I also do not want to create
> a "subset" of CRAN packages. Sorry if my explanation was confusing. There
> are internal proprietary R packages that I would like to host on a private
> repo. I like the CRAN web pages that allow a user to browse the various
> packages from a browser without having to first install them. I want to
> duplicate the process that CRAN goes through when it generate the webpages.

But for a repository of your own packages, not CRAN's packages?
Everyone who has replied so far seems to have assumed the latter ....

The scripts CRAN uses to make its 'web' area are not public.  If you can
formulate a crystal-clear message, you could ask them for a copy.

>
> Josh Bradley
>
> [[alternative HTML version deleted]]

Please do follow the posting guide: CRAN too does not appreciate HTML mail.

--
Brian D. Ripley,                  [hidden email]
Emeritus Professor of Applied Statistics, University of Oxford

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Creating a private CRAN with webpages

Dirk Eddelbuettel

On 9 June 2017 at 15:47, Prof Brian Ripley wrote:
| On 09/06/2017 07:44, Joshua Bradley wrote:
| > I'm not trying to create a mirror of the CRAN. I also do not want to create
| > a "subset" of CRAN packages. Sorry if my explanation was confusing. There
| > are internal proprietary R packages that I would like to host on a private
| > repo. I like the CRAN web pages that allow a user to browse the various
| > packages from a browser without having to first install them. I want to
| > duplicate the process that CRAN goes through when it generate the webpages.
|
| But for a repository of your own packages, not CRAN's packages?
| Everyone who has replied so far seems to have assumed the latter ....

I didn't.
 
| The scripts CRAN uses to make its 'web' area are not public.  If you can
| formulate a crystal-clear message, you could ask them for a copy.

No, write it anew, and make it prettier than the still rather awful CRAN
pages which even two years (?) after the overhaul of www.r-project.org have
the pristine and crisp look of 1998 about them. We can do better. It is 2017.

As I showed, turning packageDescription() into a paragraph of text is
trivial.  Displaying reverseDepends and what is too (per CRAN_package_db()).

So just do it.

Make it pretty enough, and hopefully one day someone will give us a prettier
version of CRAN too (beating the combination of inertia, already having
something "good enough", not enough time, and more important things to do --
really not blaming anyone here).

Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Creating a private CRAN with webpages

Zhian Kamvar-2
In reply to this post by Joshua Bradley
Hi Josh,

If you aren't opposed to using bootstrap, you could check out pkgdown: http://hadley.github.io/pkgdown/index.html <http://hadley.github.io/pkgdown/index.html>

From my experience, it works pretty well out of the box.

HTH,
Zhian

-----
Zhian N. Kamvar, Ph. D.
Postdoctoral Researcher (Everhart Lab)
Department of Plant Pathology
University of Nebraska-Lincoln

> On Jun 8, 2017, at 16:29 , Joshua Bradley <[hidden email]> wrote:
>
> Hello,
>
> I am trying to setup a private CRAN for work (behind a firewall). The best
> options available include miniCRAN
> <https://github.com/RevolutionAnalytics/miniCRAN>, drat
> <https://github.com/eddelbuettel/drat> and packrat
> <https://rstudio.github.io/packrat/>. One problem is these packages do not
> automatically generate the web pages that are on the CRAN.
>
> Examples:
> https://cran.r-project.org/web/packages/index.html
> https://cran.r-project.org/web/packages/available_packages_by_name.html
>
> Each time the CRAN adds a package, there must be an automated process in
> place to regenerate the pages again with the new package added (example - A3
> <https://cran.r-project.org/web/packages/A3/index.html>). I read somewhere
> (possibly on stackoverflow) that the CRAN html pages are statically built.
> I would like for users to be able to explore the packages in my private
> CRAN just like the public CRAN without having to open R and search for
> packages/documentation through the command line.
>
> The R Manual includes a small section
> <https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Setting-up-a-package-repository>
> on setting up a repository but it only discusses the structure of the
> directories needed to host packages. Nothing is mentioned about how the
> CRAN creates/updates the /web directory. What is the best way to
> generate/maintain the web pages for a private CRAN?
>
> P.S. Let me know if this question is best answered on one of the other
> mailing lists.
>
> Josh Bradley
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Loading...