Package httr::GET() question

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Package httr::GET() question

R help mailing list-2
Hi All:

 I hav been trying to go through the code for httr::GET() but it is somewhat beyond what I know.  What I am trying to find out is if all urls are automatically percent encoded,  or whether the user needs to do that.

Thanks,

-Roy

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: [hidden email] www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Package httr::GET() question

Ben Tupper-2
Hi,

Perhaps you could test it out by using httr::GET() with and without
escaping using xml2::url_escape()?

https://www.rdocumentation.org/packages/xml2/versions/1.2.2/topics/url_escape

Cheers,
Ben

On Tue, Feb 18, 2020 at 1:29 PM Roy Mendelssohn - NOAA Federal via
R-help <[hidden email]> wrote:

>
> Hi All:
>
>  I hav been trying to go through the code for httr::GET() but it is somewhat beyond what I know.  What I am trying to find out is if all urls are automatically percent encoded,  or whether the user needs to do that.
>
> Thanks,
>
> -Roy
>
> **********************
> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: [hidden email] www: https://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.



--
Ben Tupper
Bigelow Laboratory for Ocean Science
West Boothbay Harbor, Maine
http://www.bigelow.org/
https://eco.bigelow.org

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Package httr::GET() question

R help mailing list-2
Thanks.  Yes.  I did that,  it also has a verbose mode so that I could see what it was doing.  What I needed was not just escaping but strict escaping.  My memory forma number of years back was that I had issues with urlencode from base not being strict.  And of course you don't what to encode twice.

Thanks,

-Roy


> On Feb 19, 2020, at 7:08 AM, Ben Tupper <[hidden email]> wrote:
>
> Hi,
>
> Perhaps you could test it out by using httr::GET() with and without
> escaping using xml2::url_escape()?
>
> https://www.rdocumentation.org/packages/xml2/versions/1.2.2/topics/url_escape
>
> Cheers,
> Ben
>
> On Tue, Feb 18, 2020 at 1:29 PM Roy Mendelssohn - NOAA Federal via
> R-help <[hidden email]> wrote:
>>
>> Hi All:
>>
>> I hav been trying to go through the code for httr::GET() but it is somewhat beyond what I know.  What I am trying to find out is if all urls are automatically percent encoded,  or whether the user needs to do that.
>>
>> Thanks,
>>
>> -Roy
>>
>> **********************
>> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> ***Note new street address***
>> 110 McAllister Way
>> Santa Cruz, CA 95060
>> Phone: (831)-420-3666
>> Fax: (831) 420-3980
>> e-mail: [hidden email] www: https://www.pfeg.noaa.gov/
>>
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>
>
> --
> Ben Tupper
> Bigelow Laboratory for Ocean Science
> West Boothbay Harbor, Maine
> http://www.bigelow.org/
> https://eco.bigelow.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: [hidden email] www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Package httr::GET() question

Bob Rudis
curl::curl_escape() —
https://github.com/jeroen/curl/search?q=curl_escape&unscoped_q=curl_escape
— uses the underlying libcurl curl_easy_escape() which does proper
escaping b/c it's, well, curl.

{httr} uses curl::curl_escape() —
https://github.com/r-lib/httr/search?q=curl_escape&unscoped_q=curl_escape

The use it's `url-query.r` is the function compose_query().

compose_query()  is called from build_url() in url.r. handle_url()
(from handle-url.r) uses build_url().

All the "verbs" use handle_url() —
https://github.com/r-lib/httr/search?q=handle_url&unscoped_q=handle_url

So {httr} relies on the quintessential standard in URL escaping —
which is libcurl's — for all URL machinations.

-boB

On Wed, Feb 19, 2020 at 10:36 AM Roy Mendelssohn - NOAA Federal via
R-help <[hidden email]> wrote:

>
> Thanks.  Yes.  I did that,  it also has a verbose mode so that I could see what it was doing.  What I needed was not just escaping but strict escaping.  My memory forma number of years back was that I had issues with urlencode from base not being strict.  And of course you don't what to encode twice.
>
> Thanks,
>
> -Roy
>
>
> > On Feb 19, 2020, at 7:08 AM, Ben Tupper <[hidden email]> wrote:
> >
> > Hi,
> >
> > Perhaps you could test it out by using httr::GET() with and without
> > escaping using xml2::url_escape()?
> >
> > https://www.rdocumentation.org/packages/xml2/versions/1.2.2/topics/url_escape
> >
> > Cheers,
> > Ben
> >
> > On Tue, Feb 18, 2020 at 1:29 PM Roy Mendelssohn - NOAA Federal via
> > R-help <[hidden email]> wrote:
> >>
> >> Hi All:
> >>
> >> I hav been trying to go through the code for httr::GET() but it is somewhat beyond what I know.  What I am trying to find out is if all urls are automatically percent encoded,  or whether the user needs to do that.
> >>
> >> Thanks,
> >>
> >> -Roy
> >>
> >> **********************
> >> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
> >> **********************
> >> Roy Mendelssohn
> >> Supervisory Operations Research Analyst
> >> NOAA/NMFS
> >> Environmental Research Division
> >> Southwest Fisheries Science Center
> >> ***Note new street address***
> >> 110 McAllister Way
> >> Santa Cruz, CA 95060
> >> Phone: (831)-420-3666
> >> Fax: (831) 420-3980
> >> e-mail: [hidden email] www: https://www.pfeg.noaa.gov/
> >>
> >> "Old age and treachery will overcome youth and skill."
> >> "From those who have been given much, much will be expected"
> >> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
> >>
> >> ______________________________________________
> >> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained, reproducible code.
> >
> >
> >
> > --
> > Ben Tupper
> > Bigelow Laboratory for Ocean Science
> > West Boothbay Harbor, Maine
> > http://www.bigelow.org/
> > https://eco.bigelow.org
>
> **********************
> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: [hidden email] www: https://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Package httr::GET() question

R help mailing list-2
Thanks.  Yes that is what I found out.  We are switching our web service to require strict encoding  (a lot of services got caught when Apache Tomcat made a similar switch)  and are trying to see if it that will break any of our R packages before we make the switch,

As always,  appreciate all the people who took time to respond.

-Roy

> On Feb 22, 2020, at 5:23 AM, Bob Rudis <[hidden email]> wrote:
>
> curl::curl_escape() —
> https://github.com/jeroen/curl/search?q=curl_escape&unscoped_q=curl_escape
> — uses the underlying libcurl curl_easy_escape() which does proper
> escaping b/c it's, well, curl.
>
> {httr} uses curl::curl_escape() —
> https://github.com/r-lib/httr/search?q=curl_escape&unscoped_q=curl_escape
>
> The use it's `url-query.r` is the function compose_query().
>
> compose_query()  is called from build_url() in url.r. handle_url()
> (from handle-url.r) uses build_url().
>
> All the "verbs" use handle_url() —
> https://github.com/r-lib/httr/search?q=handle_url&unscoped_q=handle_url
>
> So {httr} relies on the quintessential standard in URL escaping —
> which is libcurl's — for all URL machinations.
>
> -boB
>
> On Wed, Feb 19, 2020 at 10:36 AM Roy Mendelssohn - NOAA Federal via
> R-help <[hidden email]> wrote:
>>
>> Thanks.  Yes.  I did that,  it also has a verbose mode so that I could see what it was doing.  What I needed was not just escaping but strict escaping.  My memory forma number of years back was that I had issues with urlencode from base not being strict.  And of course you don't what to encode twice.
>>
>> Thanks,
>>
>> -Roy
>>
>>
>>> On Feb 19, 2020, at 7:08 AM, Ben Tupper <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> Perhaps you could test it out by using httr::GET() with and without
>>> escaping using xml2::url_escape()?
>>>
>>> https://www.rdocumentation.org/packages/xml2/versions/1.2.2/topics/url_escape
>>>
>>> Cheers,
>>> Ben
>>>
>>> On Tue, Feb 18, 2020 at 1:29 PM Roy Mendelssohn - NOAA Federal via
>>> R-help <[hidden email]> wrote:
>>>>
>>>> Hi All:
>>>>
>>>> I hav been trying to go through the code for httr::GET() but it is somewhat beyond what I know.  What I am trying to find out is if all urls are automatically percent encoded,  or whether the user needs to do that.
>>>>
>>>> Thanks,
>>>>
>>>> -Roy
>>>>
>>>> **********************
>>>> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division
>>>> Southwest Fisheries Science Center
>>>> ***Note new street address***
>>>> 110 McAllister Way
>>>> Santa Cruz, CA 95060
>>>> Phone: (831)-420-3666
>>>> Fax: (831) 420-3980
>>>> e-mail: [hidden email] www: https://www.pfeg.noaa.gov/
>>>>
>>>> "Old age and treachery will overcome youth and skill."
>>>> "From those who have been given much, much will be expected"
>>>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>>>
>>>> ______________________________________________
>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>>
>>>
>>> --
>>> Ben Tupper
>>> Bigelow Laboratory for Ocean Science
>>> West Boothbay Harbor, Maine
>>> http://www.bigelow.org/
>>> https://eco.bigelow.org
>>
>> **********************
>> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> ***Note new street address***
>> 110 McAllister Way
>> Santa Cruz, CA 95060
>> Phone: (831)-420-3666
>> Fax: (831) 420-3980
>> e-mail: [hidden email] www: https://www.pfeg.noaa.gov/
>>
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: [hidden email] www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.