如何使在 jupyterlab 中本地創建的文件在 Kube.netes 中運行的 AWS S3 中可見

Question

我正在嘗試將數據從jupyter實驗室保存到 Kubenretes 中的 AWS S3。

jupyter lab 啟動后，我可以完美地看到來自 S3 存儲桶“testml1”的數據。

但是，如果我在 jupyter notebook 中打開一個終端並創建一些文件，這是不可見的：

1. 在 jupyter 的左側欄中可見
2. 也不在 S3

    extraConfig: |-
      c.ServerProxy.host_whitelist = ["localhost", "127.0.0.1", "rapidsai-scheduler"]
      from s3contents import S3ContentsManager
      
      from hybridcontents import HybridContentsManager
      from notebook.services.contents.largefilemanager import LargeFileManager
      c = get_config()
    
      c.NotebookApp.contents_manager_class = HybridContentsManager
      
      c.HybridContentsManager.manager_classes = {
          # Associate the root directory with an S3ContentsManager.
          # This manager will receive all requests that don"t fall under any of the
          # other managers.
          "": S3ContentsManager,
          # Associate /directory with a LargeFileManager.
      }
      
      c.HybridContentsManager.manager_kwargs = {
          # Args for root S3ContentsManager.
          "": {
              "access_key_id": "...",
              "secret_access_key": "...",
              "bucket": "testml1",
              # "root_dir": "/home/jovyan",
          },
      } 

它幾乎感覺像是終端，它正在使用的存儲與HybridContentsManager使用的存儲完全分開。 和S3ContentsManager

Answer 1

嘗試通過取消注釋# "root_dir": "/home/jovyan",行將映射的 S3 存儲桶關聯到筆記本工作目錄。 這應該使您的所有內容都映射到 S3，在該上下文中創建的新筆記本或文件將添加到您的存儲桶中。

如果您需要多個目錄（例如 S3 存儲桶等），您可以使用 JupyterHub 支持並記錄在Z2JH readthedocs中的額外存儲卷。

Answer 2

這是有效但仍然有效的配置。

是否可以將local_directory和AWS S3 內容放在一起？

      c.ServerProxy.host_whitelist = ["localhost", "127.0.0.1", "rapidsai-scheduler"]
      # ----------------
      from s3contents import S3ContentsManager
      from hybridcontents import HybridContentsManager
      from notebook.services.contents.largefilemanager import LargeFileManager
      #from IPython.html.services.contents.filemanager import FileContentsManager
      c = get_config()
      
      c.NotebookApp.contents_manager_class = HybridContentsManager
      
      c.HybridContentsManager.manager_classes = {
          # Associate the root directory with an S3ContentsManager.
          # This manager will receive all requests that don"t fall under any of the
          # other managers.
          "": S3ContentsManager,
          # Associate /directory with a LargeFileManager.
          "local_directory": LargeFileManager,
      }
      
      c.HybridContentsManager.manager_kwargs = {
          # Args for root S3ContentsManager.
          "": {
              "access_key_id": "...",
              "secret_access_key": "...",
              "bucket": "testml1",
          },
          "local_directory": {
              "root_dir": "/home/jovyan",
          },
      }