Some notes on using the Google Cloud storage.

Install client Library#

We need to install the gcloud storage client to use the storage API. For Python, use pip to install the client:

pip install google-cloud-storage

test in local env#

When we run the gcloud storage client in local PC and want to connect to our project, we need to do the authentication:

gcloud auth application-default login

Follow the instructions and finish the authentication. Then we can use the client locally for testing.

use the correct bucket name#

Note that bucket names refer to the top level folders when you click Cloud Storage --> Buckets. Under each bucket, you can define sub-folder structures. However, when we want to get a bucket, these sub-folders should not be part of the bucket name!!

check whether a file exists?#

from google.cloud import storage

storage_client = storage.Client()

bucket_name = "my-bucket"
bucket = storage_client.bucket(bucket_name)

file_path = "path/to/your_file/under/the/bucket"
blob = bucket.blob(file_path)

print(blob.exists())

ref:

• check whether a file exists in gcloud: https://stackoverflow.com/q/13525482/6064933

Save Python dict in gcloud storage?#

We need to convert the dict to string using JSON package and then use the method provided by blob to upload it. Here is a working code example:

import json
from google.cloud import storage

storage_client = storage.Client()

bucket_name = "my-bucket"
bucket = storage_client.bucket(bucket_name)

file_path = "path/to/your_file/under/the/bucket"
blob = bucket.blob(file_path)

content = {"message": "hello world", 'code': 0}
blob.upload_from_string(json.dumps(content))

ref:

• https://cloud.google.com/storage/docs/uploading-objects-from-memory

download file and convert to dict#

We can download the file as string and then convert it to the desired type:

import json
from google.cloud import storage

storage_client = storage.Client()

bucket_name = "my-bucket"
bucket = storage_client.bucket(bucket_name)

file_path = "path/to/your_file/under/the/bucket"
blob = bucket.blob(file_path)

content = blob.download_as_string()
content_real = json.loads(content)

ref:

• https://cloud.google.com/storage/docs/downloading-objects-into-memory