is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

John McKown
"I'm so confused!" Why does is.na() report TRUE for a POSIXlt date &
time of 2014-03-09 02:00:00 ?

> q
[1] "2014-03-09 02:00:00"
> is.na(q)
[1] TRUE
> as.POSIXct(q)
[1] NA
> dput(q)
structure(list(sec = 0, min = 0L, hour = 2, mday = 9L, mon = 2L,
    year = 114L, wday = 0L, yday = 67L, isdst = 0L, zone = "",
    gmtoff = NA_integer_), .Names = c("sec", "min", "hour", "mday",
"mon", "year", "wday", "yday", "isdst", "zone", "gmtoff"), class = c("POSIXlt",
"POSIXt"))
> str(q)
 POSIXlt[1:1], format: "2014-03-09 02:00:00"
>


--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

Duncan Murdoch-2
On 30/07/2014 1:08 PM, John McKown wrote:

> "I'm so confused!" Why does is.na() report TRUE for a POSIXlt date &
> time of 2014-03-09 02:00:00 ?
>
> > q
> [1] "2014-03-09 02:00:00"
> > is.na(q)
> [1] TRUE
> > as.POSIXct(q)
> [1] NA
> > dput(q)
> structure(list(sec = 0, min = 0L, hour = 2, mday = 9L, mon = 2L,
>      year = 114L, wday = 0L, yday = 67L, isdst = 0L, zone = "",
>      gmtoff = NA_integer_), .Names = c("sec", "min", "hour", "mday",
> "mon", "year", "wday", "yday", "isdst", "zone", "gmtoff"), class = c("POSIXlt",
> "POSIXt"))

I see an NA in there for the GMT offset, and no timezone.

Duncan Murdoch
> > str(q)
>   POSIXlt[1:1], format: "2014-03-09 02:00:00"
> >
>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

John McKown
On Wed, Jul 30, 2014 at 12:18 PM, Duncan Murdoch
<[hidden email]> wrote:

> On 30/07/2014 1:08 PM, John McKown wrote:
>>
>> "I'm so confused!" Why does is.na() report TRUE for a POSIXlt date &
>> time of 2014-03-09 02:00:00 ?
>>
>> > q
>> [1] "2014-03-09 02:00:00"
>> > is.na(q)
>> [1] TRUE
>> > as.POSIXct(q)
>> [1] NA
>> > dput(q)
>> structure(list(sec = 0, min = 0L, hour = 2, mday = 9L, mon = 2L,
>>      year = 114L, wday = 0L, yday = 67L, isdst = 0L, zone = "",
>>      gmtoff = NA_integer_), .Names = c("sec", "min", "hour", "mday",
>> "mon", "year", "wday", "yday", "isdst", "zone", "gmtoff"), class =
>> c("POSIXlt",
>> "POSIXt"))
>
>
> I see an NA in there for the GMT offset, and no timezone.

I should have mentioned that I tried other time stamps, generated the
same way as "q" above. They did not fail the is.na() test. I think
that is.na() is doing a as.double() somewhere in there because
as.double(q)  gives  NA as a result.

>
> Duncan Murdoch
>
>> > str(q)
>>   POSIXlt[1:1], format: "2014-03-09 02:00:00"
>> >
>>
>>
>



--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

William Dunlap
> I should have mentioned that I tried other time stamps, generated the
> same way as "q" above.

How did you generate q and in what time zone were you?  Note that 2am
on 9 March 2014 is when 'daylight savings time' started in the parts
of the US where it is observed.  Does 2am exist or do we jump from
1:59:59 to 3:00:00?


Bill Dunlap
TIBCO Software
wdunlap tibco.com


On Wed, Jul 30, 2014 at 10:42 AM, John McKown
<[hidden email]> wrote:

> On Wed, Jul 30, 2014 at 12:18 PM, Duncan Murdoch
> <[hidden email]> wrote:
>> On 30/07/2014 1:08 PM, John McKown wrote:
>>>
>>> "I'm so confused!" Why does is.na() report TRUE for a POSIXlt date &
>>> time of 2014-03-09 02:00:00 ?
>>>
>>> > q
>>> [1] "2014-03-09 02:00:00"
>>> > is.na(q)
>>> [1] TRUE
>>> > as.POSIXct(q)
>>> [1] NA
>>> > dput(q)
>>> structure(list(sec = 0, min = 0L, hour = 2, mday = 9L, mon = 2L,
>>>      year = 114L, wday = 0L, yday = 67L, isdst = 0L, zone = "",
>>>      gmtoff = NA_integer_), .Names = c("sec", "min", "hour", "mday",
>>> "mon", "year", "wday", "yday", "isdst", "zone", "gmtoff"), class =
>>> c("POSIXlt",
>>> "POSIXt"))
>>
>>
>> I see an NA in there for the GMT offset, and no timezone.
>
> I should have mentioned that I tried other time stamps, generated the
> same way as "q" above. They did not fail the is.na() test. I think
> that is.na() is doing a as.double() somewhere in there because
> as.double(q)  gives  NA as a result.
>
>>
>> Duncan Murdoch
>>
>>> > str(q)
>>>   POSIXlt[1:1], format: "2014-03-09 02:00:00"
>>> >
>>>
>>>
>>
>
>
>
> --
> There is nothing more pleasant than traveling and meeting new people!
> Genghis Khan
>
> Maranatha! <><
> John McKown
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

Jeff Newmiller
In reply to this post by John McKown
Isn't that a timestamp that doesn't exist in standard US timezones? Maybe use a timezone that doesn't consider daylight savings (like "Etc/GMT+5")?
---------------------------------------------------------------------------
Jeff Newmiller                        The     .....       .....  Go Live...
DCN:<[hidden email]>        Basics: ##.#.       ##.#.  Live Go...
                                      Live:   OO#.. Dead: OO#..  Playing
Research Engineer (Solar/Batteries            O.O#.       #.O#.  with
/Software/Embedded Controllers)               .OO#.       .OO#.  rocks...1k
---------------------------------------------------------------------------
Sent from my phone. Please excuse my brevity.

On July 30, 2014 10:08:58 AM PDT, John McKown <[hidden email]> wrote:

>"I'm so confused!" Why does is.na() report TRUE for a POSIXlt date &
>time of 2014-03-09 02:00:00 ?
>
>> q
>[1] "2014-03-09 02:00:00"
>> is.na(q)
>[1] TRUE
>> as.POSIXct(q)
>[1] NA
>> dput(q)
>structure(list(sec = 0, min = 0L, hour = 2, mday = 9L, mon = 2L,
>    year = 114L, wday = 0L, yday = 67L, isdst = 0L, zone = "",
>    gmtoff = NA_integer_), .Names = c("sec", "min", "hour", "mday",
>"mon", "year", "wday", "yday", "isdst", "zone", "gmtoff"), class =
>c("POSIXlt",
>"POSIXt"))
>> str(q)
> POSIXlt[1:1], format: "2014-03-09 02:00:00"
>>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

John McKown
In reply to this post by William Dunlap
On Wed, Jul 30, 2014 at 12:54 PM, William Dunlap <[hidden email]> wrote:
>> I should have mentioned that I tried other time stamps, generated the
>> same way as "q" above.
>
> How did you generate q and in what time zone were you?

I got it from an MS-SQL data base which is maintained by some
closed-source vendor software. But I manipulate the data in the SELECT
before sending it to R via ODBC. I need to double check the raw data
in the data base.

> Note that 2am
> on 9 March 2014 is when 'daylight savings time' started in the parts
> of the US where it is observed.  Does 2am exist or do we jump from
> 1:59:59 to 3:00:00?

Hum, that hadn't occurred to me. I need to see what is in the DB.

But I think you have found my problem. If I force the timezone to be
GMT, then the problem disappears. So that is what I'll do with this
data.

>
>
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com

--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

John McKown
OK, daylight saving time, may be be cursed, is definitely my problem.
Strangely, the DB has data for time 0000, 0100, 0200, 0400, 0500, ...
2300. It is closed source, so I have _no_ idea how this happened. And
I cannot report bugs because it is no longer supported. The encoding
is a 4 character (digit) field of the form HHMM. Why, I don't know.

But I have a work around which satisfies me and lets me get something
for the boss.

On Wed, Jul 30, 2014 at 1:07 PM, John McKown
<[hidden email]> wrote:

> On Wed, Jul 30, 2014 at 12:54 PM, William Dunlap <[hidden email]> wrote:
>>> I should have mentioned that I tried other time stamps, generated the
>>> same way as "q" above.
>>
>> How did you generate q and in what time zone were you?
>
> I got it from an MS-SQL data base which is maintained by some
> closed-source vendor software. But I manipulate the data in the SELECT
> before sending it to R via ODBC. I need to double check the raw data
> in the data base.
>
>> Note that 2am
>> on 9 March 2014 is when 'daylight savings time' started in the parts
>> of the US where it is observed.  Does 2am exist or do we jump from
>> 1:59:59 to 3:00:00?
>
> Hum, that hadn't occurred to me. I need to see what is in the DB.
>
> But I think you have found my problem. If I force the timezone to be
> GMT, then the problem disappears. So that is what I'll do with this
> data.
>
>>
>>
>> Bill Dunlap
>> TIBCO Software
>> wdunlap tibco.com
>
> --
> There is nothing more pleasant than traveling and meeting new people!
> Genghis Khan
>
> Maranatha! <><
> John McKown



--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

William Dunlap
In reply to this post by John McKown
I meant what R commands did you use to change the database's version
of the time/date object to the R version?
Bill Dunlap
TIBCO Software
wdunlap tibco.com


On Wed, Jul 30, 2014 at 11:07 AM, John McKown
<[hidden email]> wrote:

> On Wed, Jul 30, 2014 at 12:54 PM, William Dunlap <[hidden email]> wrote:
>>> I should have mentioned that I tried other time stamps, generated the
>>> same way as "q" above.
>>
>> How did you generate q and in what time zone were you?
>
> I got it from an MS-SQL data base which is maintained by some
> closed-source vendor software. But I manipulate the data in the SELECT
> before sending it to R via ODBC. I need to double check the raw data
> in the data base.
>
>> Note that 2am
>> on 9 March 2014 is when 'daylight savings time' started in the parts
>> of the US where it is observed.  Does 2am exist or do we jump from
>> 1:59:59 to 3:00:00?
>
> Hum, that hadn't occurred to me. I need to see what is in the DB.
>
> But I think you have found my problem. If I force the timezone to be
> GMT, then the problem disappears. So that is what I'll do with this
> data.
>
>>
>>
>> Bill Dunlap
>> TIBCO Software
>> wdunlap tibco.com
>
> --
> There is nothing more pleasant than traveling and meeting new people!
> Genghis Khan
>
> Maranatha! <><
> John McKown

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

John McKown
Probably not the best, but here:

con <- odbcConnect("BMC");
timezone <- Sys.timezone();
#
query=paste0("select CONVERT(smalldatetime,Int_Start_Date,11) as
Int_Start_Date,",
             " CONVERT(smalldatetime,CASE WHEN Int_Start_Time is NULL
then '00:00' ",
             "else
LEFT(Int_Start_Time,2)+':'+SUBSTRING(Int_Start_Time,3,2) end +",
             "':00', 14) as Int_Start_Time",
             ", Int_duration, RTRIM(INTTYPE) AS INTTYPE",
             ", RTRIM(Int_descr) AS Int_descr",
             ", RTRIM(INTSUBT) as INTSUBT",
             ", INDEXX, RTRIM(Label) AS Label",
             ", RTRIM(CHANGED) AS CHANGED",
             ", RTRIM(ALERT) AS ALERT",
             ", RTRIM(RELEASE) AS RELEASE",
             " FROM CPINTVL where Int_Start_Date BETWEEN '",
             startDateChar,"' and '",endDateChar,"'",
             "AND INTTYPE='M'"
);
cpintvl <- sqlQuery(con,
            query,
            stringsAsFactors=FALSE,
            as.is=TRUE);
#
# properly combine start date and time because I couldn't figure out how to
# get MS-SQL to do it for me.
cpintvl$Int_Start <- strptime(paste0(substr(cpintvl$Int_Start_Date,1,11),

substr(cpintvl$Int_Start_Time,12,19)),"%Y-%m-%d %H:%M:%S");

So the actual value was created with the strptime() call at the end.
The rest is just in character form. The Int_Start_Date in the DB is in
yy/mm/dd format as a character field. Int_Start_Time is HHMM as a
character field with leading zeros. The return value from the SELECT
has Start_Int_Date formatted in yyyy-mm-dd as a character string.
Start_Int_Time formatted in hh:mm:ss as a character string.

In any case, you have accurately explained my foolishness. I keep
forgetting about DST because I record _everything_ in my personal DBs
in UTC.

On Wed, Jul 30, 2014 at 1:23 PM, William Dunlap <[hidden email]> wrote:

> I meant what R commands did you use to change the database's version
> of the time/date object to the R version?
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com
>
>
> On Wed, Jul 30, 2014 at 11:07 AM, John McKown
> <[hidden email]> wrote:
>> On Wed, Jul 30, 2014 at 12:54 PM, William Dunlap <[hidden email]> wrote:
>>>> I should have mentioned that I tried other time stamps, generated the
>>>> same way as "q" above.
>>>
>>> How did you generate q and in what time zone were you?
>>
>> I got it from an MS-SQL data base which is maintained by some
>> closed-source vendor software. But I manipulate the data in the SELECT
>> before sending it to R via ODBC. I need to double check the raw data
>> in the data base.
>>
>>> Note that 2am
>>> on 9 March 2014 is when 'daylight savings time' started in the parts
>>> of the US where it is observed.  Does 2am exist or do we jump from
>>> 1:59:59 to 3:00:00?
>>
>> Hum, that hadn't occurred to me. I need to see what is in the DB.
>>
>> But I think you have found my problem. If I force the timezone to be
>> GMT, then the problem disappears. So that is what I'll do with this
>> data.
>>
>>>
>>>
>>> Bill Dunlap
>>> TIBCO Software
>>> wdunlap tibco.com
>>
>> --
>> There is nothing more pleasant than traveling and meeting new people!
>> Genghis Khan
>>
>> Maranatha! <><
>> John McKown



--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: is.na() == TRUE for POSIXlt time / date of "2014-03-09 02:00:00"

arun kirshna
In reply to this post by John McKown
Not able to reproduce the problem.
str(q)
# POSIXlt[1:1], format: "2014-03-09 02:00:00"
 is.na(q)
#[1] FALSE
sessionInfo()
R version 3.1.0 (2014-04-10)
Platform: x86_64-unknown-linux-gnu (64-bit)
A.K.




On Wednesday, July 30, 2014 1:10 PM, John McKown <[hidden email]> wrote:
"I'm so confused!" Why does is.na() report TRUE for a POSIXlt date &
time of 2014-03-09 02:00:00 ?

> q
[1] "2014-03-09 02:00:00"
> is.na(q)
[1] TRUE
> as.POSIXct(q)
[1] NA
> dput(q)
structure(list(sec = 0, min = 0L, hour = 2, mday = 9L, mon = 2L,
    year = 114L, wday = 0L, yday = 67L, isdst = 0L, zone = "",
    gmtoff = NA_integer_), .Names = c("sec", "min", "hour", "mday",
"mon", "year", "wday", "yday", "isdst", "zone", "gmtoff"), class = c("POSIXlt",
"POSIXt"))
> str(q)
POSIXlt[1:1], format: "2014-03-09 02:00:00"
>


--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.