signed urls do not work in Cloud Run with django-storages

See original GitHub issue

Using Cloud Run the default credentials do not allow signing of urls. The error is swallowed in most use cases and the file filed just returns a None in serializers or in django admin.

The error is confusing because everything works fine on the upload side. Even worse, if you use the same service account and run your container locally - it works fine. Since the credential is provided in the different manner.

To recreate:

setup Django project in Cloud Run with Media objects in stored using GCS.
Create file in Django admin (the file field comes back as None, but the file is in the GCS bucket)
On editing a file in Django admin, the error is displayed:

Exception Type: | AttributeError
-- | --
you need a private key to sign credentials.the credentials you are currently using <class 'google.auth.compute_engine.credentials.Credentials'> just contains a token. see https://googleapis.dev/python/google-api-core/latest/auth.html#setting-up-a-service-account for more details.
/usr/local/lib/python3.8/site-packages/google/cloud/storage/_signing.py, line 51, in ensure_signed_credentials

I am still trying to figure out the best work around, but I wanted to add this in case anyone else runs into the error. This maybe a documentation update or a change to not swallow the error.

Issue Analytics

State:
Created 3 years ago
Reactions:9
Comments:9 (1 by maintainers)

Top GitHub Comments

5reactions

danfairscommented, Mar 12, 2021

Our workaround is as follows:

import datetime

from django.core.cache import cache
from django.utils.deconstruct import deconstructible

import google.auth
import google.auth.compute_engine
import google.auth.transport.requests
from djangoapps.core.context import get_context
# We'll take it on the chin if this moves
from google.cloud.storage.blob import _quote
from storages.backends.gcloud import GoogleCloudStorage
from storages.utils import clean_name


@deconstructible
class GoogleCloudStorageAccessToken(GoogleCloudStorage):
    CACHE_KEY = "GoogleCloudStorageAccessToken.signing_extras"

    def url(self, name):
        """
        Return public url or a signed url for the Blob.
        This DOES NOT check for existance of Blob - that makes codes too slow
        for many use cases.

        We override this to provide an extra information to url signing, so we don't need to have a private key
        available. This is a workaround for https://github.com/jschneier/django-storages/issues/941.
        """
        name = self._normalize_name(clean_name(name))
        blob = self.bucket.blob(name)
        no_signed_url = self.default_acl == "publicRead" or not self.querystring_auth

        if not self.custom_endpoint and no_signed_url:
            return blob.public_url
        elif no_signed_url:
            return "{storage_base_url}/{quoted_name}".format(
                storage_base_url=self.custom_endpoint,
                quoted_name=_quote(name, safe=b"/~"),
            )
        elif not self.custom_endpoint:
            return blob.generate_signed_url(self.expiration, **self.signed_url_extra())
        else:
            return blob.generate_signed_url(
                expiration=self.expiration,
                api_access_endpoint=self.custom_endpoint,
                **self.signed_url_extra()
            )

    def signed_url_extra(self):
        value = cache.get(self.CACHE_KEY)
        if value is not None:
            expiry, extra = value
            if expiry > datetime.datetime.utcnow():
                return extra

        credentials, project_id = google.auth.default()
        auth_req = google.auth.transport.requests.Request()
        credentials.refresh(auth_req)
        extra = {
            "service_account_email": credentials.service_account_email,
            "access_token": credentials.token,
            "credentials": credentials,
        }

        cache.set(self.CACHE_KEY, (credentials.expiry, extra))
        return extra

You should obviously then use this class as a replacement for the GoogleCloudStorage base class (we set up storages explicitly on a case-by-case basis, so this is fairly straightforard for us).

2reactions

therefromherecommented, Mar 10, 2022

I know the OP wanted to use signed URLs, just a note we hit this issue accidentally because we were missing GS_DEFAULT_ACL from our settings.py, so were accidentally enabling signed URLs.

GS_DEFAULT_ACL = 'publicRead’