无法在具有权限的文件夹中的HDFS上创建登台目录

Question

There are couple of folders in the root dir of HDFS:

• dir1
  • subdir1
    • table1
    • table2
  • subdir2
• dir2
  • subdir1
    • table1
    • table2
• dir3

They all have subfolders that contain different Parquet files that are queried with Hive. I can't load one of the subfolders (for example table1 inside dir2) even though the permissions look ok to me, I get the EXECUTE error when trying to load it. The code is running in a Jupyter notebook. Users are organized in groups.

I've added rwx permissions for the directory in question to the group by using the following command:

hdfs dfs -setfacl -R -m group:user_group:rwx /dir2/subdir2

The error I'm getting looks like this:

Cannot create staging directory 'hdfs://server:8020/dir2/subdir1/table1/.hive-staging_hive_2019-08-01_13-04-22': Permission denied: user=username, access=EXECUTE, inode="/dir2":hdfs:supergroup:drwxrwx---

I've added read and execute permissions on dir2 to the user group but the error persists. It looks to me from this error that somehow the default permissions are applied and they are ---

So, to summarize; group has read and execute privileges on the root dir, and read, write and execute privileges on the table directories, but it keeps failing with permissions for root directory.

This is how the permissions look:

# file: /dir2
# owner: hdfs
# group: supergroup
user::rwx
user:some_group1:r-x
group::---
group:some_group2:rwx
group:user_group:r-x
group:hive:rwx
group:some_group3:r-x
group:some_group4:r-x
mask::rwx
other::---
default:user::rwx
default:user:some_group1:r-x
default:group::---
default:group:some_group2:rwx
default:group:hive:rwx
default:group:some_group3:r-x
default:group:some_group4:r-x
default:mask::rwx
default:other::---


# file: /dir2/subdir1/table1
# owner: some_user
# group: supergroup
user::rwx
user:some_group1:r-x
group::---
group:some_group2:rwx
group:user_group:rwx
group:hive:rwx
group:some_group3:r-x
group:some_group4:rwx
mask::rwx
other::---
default:user::rwx
default:user:some_group1:r-x
default:group::---
default:group:some_group2:rwx
default:group:user_group:rwx
default:group:hive:rwx
default:group:some_group3:r-x
default:group:some_group4:rwx
default:mask::rwx
default:other::---

Answer 1

The problem was eventually solved by creating new directories that replaced the old ones. The new directories were created with the correct user and credentials. For example, I created subdir1_new, moved the data there, renamed subdir1 to subdir1_old and renamed subdir1_new to subdir1. Not a lot of folders were affected by this issue so it didn't take a long time.

I know it's not the actual solution, but I couldn't figure out what exactly was happening and this workaround did the trick.