How to use Cloud Scheduler to store data into Cloud Storage?

Question

I've created a job in Google Cloud Scheduler to download data from my demo app using HTTP GET it seemed to run successfully. My question is where did it store that data? And how can I store it into Google Cloud Storage? Below is a screenshot of my job:

I'm new to Google Cloud and working with a free trial account. Please advise.

Answer 1

Cloud Scheduler does not process data. The data returned by your example request is discarded.

Write a Cloud Function scheduled by Cloud Scheduler. There are other services such as Firebase and Cloud Run that work well also.

Answer 2

What you're trying to do is that you're trying to create a scheduler job that calls a GET request, and Cloud Scheduler will do exactly that, except the part where data is stored. Since Cloud Scheduler is a managed-cron service, it doesn't matter if the URL returns data. Cloud Scheduler will call the endpoint on a timely manner and that's it. Data returned by the request will be discarded, as mentioned by John Hanley.

What you can do is to integrate scheduling with Cloud Functions, but to be clear, your app needs to do the following first:

1. Download from external link and save the object to /tmp within the function.

The rest of the file system is read-only, and /tmp is the only writeable part. Any files saved in /tmp are stored within the function's memory. You can clear /tmp for every successful upload to GCS.

2. Upload the file to Cloud Storage (using Cloud Storage client library).

Now that your app is capable of storing and uploading data, then you can make a decision where to deploy it.

You can deploy your code using Firebase CLI and use scheduled functions. The advantage is that Pub/Sub and Cloud Scheduler is taken care of automatically and configuration is done on your code. Downside is that you are limited to Node Runtime compared to GCP Cloud Functions, where there are many different programming languages available. Learn more about scheduled functions .

Second, you can deploy through gcloud CLI but for this, you need to setup Pub/Sub notifications and Cloud Scheduler. You can check more about this by navigating to this link .