How to download those files from github?

Question

I use windows and the Anaconda distribution of Python and git bash to read.sh files. I am not able to install files here: https://github.com/TalwalkarLab/leaf/tree/master/data/femnist

I cloned the leaf folder in python using git bash and the command git clone https://github.com/TalwalkarLab/leaf.git

As the readme.md file in /femnist suggests, I tried to run preprocess.sh (firstly with no specific arguments)

I firstly had troubles because the shell didn't know what "python3" was, so I made a copy of "python.exe" and renamed it "python3.exe".

When I go in git bash with cd {path where we can find preprocess.sh}, I try to run preprocess.sh and I obtain:

IS234906+admin-local@IS234906 MINGW64 ~/Desktop/CEA/femnist/leaf/data/femnist (master)
$ ./preprocess.sh
------------------------------
extracting file directories of images
Traceback (most recent call last):
  File "get_file_dirs.py", line 27, in <module>
    classes = os.listdir(class_dir)
FileNotFoundError: [WinError 3] The specified path cannot be found: 'C:\\Users\\admin-local\\Desktop\\CEA\\femnist\\leaf\\data\\femnist\\data\\raw_data\\by_class'
finished extracting file directories of images
------------------------------
calculating image hashes
Traceback (most recent call last):
  File "get_hashes.py", line 15, in <module>
    class_file_dirs = util.load_obj(cfd)
  File "C:\Users\admin-local\Desktop\CEA\femnist\leaf\data\utils\util.py", line 10, in load_obj
    with open(name + '.pkl', 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\admin-local\\Desktop\\CEA\\femnist\\leaf\\data\\femnist\\data\\intermediate\\class_file_dirs.pkl'
finished calculating image hashes
------------------------------
assigning class labels to write images
Traceback (most recent call last):
  File "match_hashes.py", line 14, in <module>
    class_file_hashes = util.load_obj(cfhd) # each elem is (class, file dir, hash)
  File "C:\Users\admin-local\Desktop\CEA\femnist\leaf\data\utils\util.py", line 10, in load_obj
    with open(name + '.pkl', 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\admin-local\\Desktop\\CEA\\femnist\\leaf\\data\\femnist\\data\\intermediate\\class_file_hashes.pkl'
finished assigning class labels to write images
------------------------------
grouping images by writer
Traceback (most recent call last):
  File "group_by_writer.py", line 13, in <module>
    write_class = util.load_obj(wwcd)
  File "C:\Users\admin-local\Desktop\CEA\femnist\leaf\data\utils\util.py", line 10, in load_obj
    with open(name + '.pkl', 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\admin-local\\Desktop\\CEA\\femnist\\leaf\\data\\femnist\\data\\intermediate\\write_with_class.pkl'
finished grouping images by writer
------------------------------
converting data to .json format
Traceback (most recent call last):
  File "data_to_json.py", line 43, in <module>
    writers = util.load_obj(by_writer_dir)
  File "C:\Users\admin-local\Desktop\CEA\femnist\leaf\data\utils\util.py", line 10, in load_obj
    with open(name + '.pkl', 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\admin-local\\Desktop\\CEA\\femnist\\leaf\\data\\femnist\\data\\intermediate\\images_by_writer.pkl'
finished converting data to .json format
------------------------------
calculating JSON file checksums
checksums written to meta/dir-checksum.md5

If someone has a clue about what should I do, it would help me a lot.

Thank you all by advance and have a nice day !

Answer 1

First thing I notice is that you said your shell didn't know what python3 was. That means you need to actually install python3 on your computer. Renaming python.exe is not a valid workaround. In your terminal you will need to run python3 --version . If this doesn't return a python3 version number to you, then you know you do not have python3 installed. Python 3.7 can be installed from the windows store or from https://www.python.org/downloads/windows/

After that, and re-cloning the entire repo with git clone https://github.com/TalwalkarLab/leaf.git , you will need to install the packages mentioned in the readme file using the pip3 install -r requirements.txt command with your terminal in the repo's root directory.

Whence you've done this high-level install, now you use cd data/femnist to enter the directory of the dataset. Finally, run the command bash./preprocess.sh .

I followed those exact steps (however, I am on Mac OSX), and was able to download all the data this way.

Please let me know the results of trying my solution and I will be glad to provide any more insight if I can!