从jupyter执行Google Cloud Shell命令

Question

I need to upload file from FTP to Google Cloud Storage on daily basis. I have managed to make it in Python by downloading file from FTP and uploading to Google Cloud Storage, but it seems too heavy. So, i am curious if it is possible to make it by executing commands in Google Cloud Shell, or is there any other possible solutions to make it in more optimal way.

from gcloud import storage
from oauth2client.service_account import ServiceAccountCredentials
import os

ftp = ftplib.FTP("ftp_url")
ftp.login('login', 'password')
ftp.cwd("/")


with open('file', 'wb') as f:
    ftp.retrbinary('RETR ' + 'file', f.write)

credentials = ServiceAccountCredentials.from_json_keyfile_dict(
    credentials_dict
)

client = storage.Client(credentials=credentials, project='project_id')

bucket = client.get_bucket('bucket')
blob = bucket.blob('file')
blob.upload_from_filename('file')

Answer 1

Maybe you could use "Cloud Storage FUSE" to mount Cloud Storage buckets as file systems directly in your FTP server to help in the issue reported in the description of this question.

Now, answering the title of the question, you can use this code as an example:

Pip Install:

!pip install google-cloud
!pip install google-api-python-client
!pip install oauth2client
!pip install google-cloud-bigquery

Code:

import subprocess
import logging
from google.cloud import storage

logger = logging.Logger('catch_all')

def execute_bash(parameters):
    try:
        return subprocess.check_output(parameters)
    except Exception as e: 
       logger.error(e) 
       logger.error('ERROR: Looking in jupyter console for more information')

def example_list_bucket_gcs():
    list_bucket = execute_bash(['gsutil', 'ls']).decode("utf-8").split('\n')
    for bucket in list_bucket:
        print(bucket)

def example_list_bucket_api(client_gcs):
    list_bucket = client_gcs.list_buckets()
    for bucket in list_bucket:
        print(bucket.name)

JSON_FILE_NAME = 'sa_bq.json'
client_gcs = storage.Client.from_service_account_json(JSON_FILE_NAME)
example_list_bucket_api(client_gcs)
example_list_bucket_gcs()