Request for help about running a loop and reading .NC files

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Request for help about running a loop and reading .NC files

Bhaskar Mitra
Hello Everyone,

I have written a loop which reads hundreds of .nc files
and extract information from each .nc file and
exports that corresponding information as a csv file.

The loop works fine until it encounters a .nc file which it cannot read
and the loop stops.

 I would appreciate if anyone can suggest how can I modify
the loop, whereby the loop will continue to run by bypassing those
particular
files which it cannot read or if any particular file has an error.

In the end, I was also hoping to modify the loop such that it
will generate a report which will inform me which
files were not read by the loop.  The codes are given below

Thanks for your help,

Regards,
Bhaskar Mitra


#_------------------------------------------------------------------


library(ncdf4)
library(reshape2)
library(dplyr)
library(stringr)


setwd("directory path")

Output <- "directory path"

flist <- list.files(path ="NCFiles/", pattern = "^.*\\.(nc|NC|Nc|Nc)$")

for (i in 1: length(flist))
{

  nc <- nc_open(paste0("NCFiles/",flist[i]))
  mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
  nc_close(nc)

  mean_chl_df <- melt(mean1)
  trial.table.df <-as.data.frame(mean_chl_df)

write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))

}

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Request for help about running a loop and reading .NC files

Ivan Krylov
On Mon, 23 Mar 2020 16:16:42 -0700
Bhaskar Mitra <[hidden email]> wrote:

> I would appreciate if anyone can suggest how can I modify
>the loop, whereby the loop will continue to run by bypassing those
>particular
>files which it cannot read or if any particular file has an error.

See ?try and ?tryCatch. With `try`, you can use inherits(x,
'try-error') to check whether execution failed. With `tryCatch` you can
choose your own value to return instead of the error object.

--
Best regards,
Ivan

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Request for help about running a loop and reading .NC files

Bert Gunter-2
In reply to this post by Bhaskar Mitra
?tryCatch

See also:
https://www.r-bloggers.com/error-handling-in-r/


Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )

On Tue, Mar 24, 2020 at 5:13 AM Bhaskar Mitra <[hidden email]> wrote:

>
> Hello Everyone,
>
> I have written a loop which reads hundreds of .nc files
> and extract information from each .nc file and
> exports that corresponding information as a csv file.
>
> The loop works fine until it encounters a .nc file which it cannot read
> and the loop stops.
>
>  I would appreciate if anyone can suggest how can I modify
> the loop, whereby the loop will continue to run by bypassing those
> particular
> files which it cannot read or if any particular file has an error.
>
> In the end, I was also hoping to modify the loop such that it
> will generate a report which will inform me which
> files were not read by the loop.  The codes are given below
>
> Thanks for your help,
>
> Regards,
> Bhaskar Mitra
>
>
> #_------------------------------------------------------------------
>
>
> library(ncdf4)
> library(reshape2)
> library(dplyr)
> library(stringr)
>
>
> setwd("directory path")
>
> Output <- "directory path"
>
> flist <- list.files(path ="NCFiles/", pattern = "^.*\\.(nc|NC|Nc|Nc)$")
>
> for (i in 1: length(flist))
> {
>
>   nc <- nc_open(paste0("NCFiles/",flist[i]))
>   mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
>   nc_close(nc)
>
>   mean_chl_df <- melt(mean1)
>   trial.table.df <-as.data.frame(mean_chl_df)
>
> write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))
>
> }
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Request for help about running a loop and reading .NC files

R help mailing list-2
In reply to this post by Bhaskar Mitra
 Hey Bhaskar,

Dave Pierce here, the author and maintainer of the ncdf4 package. What
platform are you working on? If you work on Linux, I'd be happy to send you
a new source tar version of the ncdf4 package I've made that handles this
case. It now has a settable parameter in nc_open called "return_on_error",
and if you set that to TRUE, then nc_open always returns. If no error was
encountered, then the return list element $error is FALSE, and if an error
was encountered, $error is set to TRUE. So you could do this:

for( ifile in 1:nfiles ) {
    nc = nc_open( filename[ifile], return_on_error=TRUE )
    if( nc$error == FALSE ) {
        ... do your processing here ...
        }
  else
        print(paste("Error encountered, skipping file", filename[ifile]))

Regards,

--Dave

On Tue, Mar 24, 2020 at 5:13 AM Bhaskar Mitra <[hidden email]>
wrote:

> Hello Everyone,
>
> I have written a loop which reads hundreds of .nc files
> and extract information from each .nc file and
> exports that corresponding information as a csv file.
>
> The loop works fine until it encounters a .nc file which it cannot read
> and the loop stops.
>
>  I would appreciate if anyone can suggest how can I modify
> the loop, whereby the loop will continue to run by bypassing those
> particular
> files which it cannot read or if any particular file has an error.
>
> In the end, I was also hoping to modify the loop such that it
> will generate a report which will inform me which
> files were not read by the loop.  The codes are given below
>
> Thanks for your help,
>
> Regards,
> Bhaskar Mitra
>
>
> #_------------------------------------------------------------------
>
>
> library(ncdf4)
> library(reshape2)
> library(dplyr)
> library(stringr)
>
>
> setwd("directory path")
>
> Output <- "directory path"
>
> flist <- list.files(path ="NCFiles/", pattern = "^.*\\.(nc|NC|Nc|Nc)$")
>
> for (i in 1: length(flist))
> {
>
>   nc <- nc_open(paste0("NCFiles/",flist[i]))
>   mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
>   nc_close(nc)
>
>   mean_chl_df <- melt(mean1)
>   trial.table.df <-as.data.frame(mean_chl_df)
>
>
> write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))
>
> }
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


--

-------------------------------------------------------------------
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography
(858) 534-8276 (voice)  /  (858) 534-8561 (fax)    [hidden email]
-------------------------------------------------------------------

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.