TypeError: invalid path or file error even when the file exists

Question

My code raised "invalid path or file" error even when the file exists. When I check the list of files in the path, it shows "permission denied" even though I'm root.

import rasterio

sample = pd.read_csv(os.path.join(config.BASE_PATH, "sample_submission.csv"))
test_images = glob.glob(os.path.join(config.BASE_PATH + "test_images", "**", "*.tiff"), recursive=True)

class HuBMAPDataset:
    def __init__(self, idx, sz=sz, reduce=reduce):
        self.data = rasterio.open(test_images, transform = identity, num_threads='all_cpus')    

for idx,row in tqdm(sample.iterrows(),total=len(sample)):
    idx = str(row['id'])
    ds = HuBMAPDataset(idx)

Traceback:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [89], in <cell line: 2>()
      2 for idx,row in tqdm(sample.iterrows(),total=len(sample)):
      3     idx = str(row['id'])
----> 4     ds = HuBMAPDataset(idx)
      5     #rasterio cannot be used with multiple workers
      6     dl = DataLoader(ds,bs,num_workers=0,shuffle=False,pin_memory=True)

Input In [85], in HuBMAPDataset.__init__(self, idx, sz, reduce)
     14 def __init__(self, idx, sz=sz, reduce=reduce):
     15     #self.data = rasterio.open(os.path.join(config.BASE_PATH, test_images,idx+'.tiff'), transform = identity, num_threads='all_cpus')      
---> 16     self.data = rasterio.open(test_images, transform = identity, num_threads='all_cpus')     
     18     # some images have issues with their format 
     19     # and must be saved correctly before reading with rasterio
     20     if self.data.count != 3:

File ~/anaconda3/lib/python3.9/site-packages/rasterio/env.py:442, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    439     session = DummySession()
    441 with env_ctor(session=session):
--> 442     return f(*args, **kwds)

File ~/anaconda3/lib/python3.9/site-packages/rasterio/__init__.py:189, in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    183 if not isinstance(fp, str):
    184     if not (
    185         hasattr(fp, "read")
    186         or hasattr(fp, "write")
    187         or isinstance(fp, (os.PathLike, MemoryFile, FilePath))
    188     ):
--> 189         raise TypeError("invalid path or file: {0!r}".format(fp))
    190 if mode and not isinstance(mode, str):
    191     raise TypeError("invalid mode: {0!r}".format(mode))

TypeError: invalid path or file: ['./input/hubmap-organ-segmentation/test_images/10078.tiff']

File exists in path but permission denied.

!./input/hubmap-organ-segmentation/test_images/10078.tiff
/bin/bash: ./input/hubmap-organ-segmentation/test_images/10078.tiff: Permission denied

Answer 1

This is because you do not have permissions to either read or write to that file/directory.

Try to update the permission of the directory by running chmod -R 660 folder_name: . This command will recursively update the permissions to read and write and then try again.

Also, there is a possibility, that with the root user you do have permissions, but the anaconda process you run, has been started with a different user. Try to run anaconda with root user and test, although it is not recommended. Would be much better to fix the permissions for your user. It's safer.