o ˷eّ@sdZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl m Zddl mZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZGdd d e Z!Gd d d e!Z"Gd d d e!Z#Gddde!ej$Z$ddZ%ddZ&ddZ'ddZ(ddZ)dS)zVirtual bases classes for uploading media via Google APIs. Supported here are: * simple (media) uploads * multipart uploads that contain both metadata and a small file as payload * resumable uploads (with metadata as well) N)_async_resumable_media)_helpers)_upload)common)_CONTENT_TYPE_HEADER_CONTENT_RANGE_TEMPLATE_RANGE_UNKNOWN_TEMPLATE_EMPTY_RANGE_TEMPLATE_BOUNDARY_FORMAT_MULTIPART_SEP_CRLF_MULTIPART_BEGIN_RELATED_HEADER_BYTES_RANGE_RE_STREAM_ERROR_TEMPLATE_POST_PUT!_UPLOAD_CHECKSUM_MISMATCH_MESSAGE0_UPLOAD_METADATA_NO_APPROPRIATE_CHECKSUM_MESSAGEc@sReZdZdZdddZeddZddZed d Z ed d Z ed dZ dS) UploadBaseaBase class for upload helpers. Defines core shared behavior across different upload types. Args: upload_url (str): The URL where the content will be uploaded. headers (Optional[Mapping[str, str]]): Extra headers that should be sent with the request, e.g. headers for encrypted data. Attributes: upload_url (str): The URL where the content will be uploaded. NcCs,||_|dur i}||_d|_t|_dS)NF) upload_url_headers _finishedr RetryStrategy_retry_strategy)selfrheadersr\/var/www/ideatree/venv/lib/python3.10/site-packages/google/_async_resumable_media/_upload.py__init__Fs zUploadBase.__init__cC|jS)z2bool: Flag indicating if the upload has completed.)rrrrrfinishedNzUploadBase.finishedcCs d|_t|tjjf|jdS)aProcess the response from an HTTP request. This is everything that must be done after a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. Args: response (object): The HTTP response object. Raises: ~google.resumable_media.common.InvalidResponse: If the status code is not 200. .. _sans-I/O: https://sans-io.readthedocs.io/ TN)rrrequire_status_codehttpclientOK_get_status_coderresponserrr_process_responseSszUploadBase._process_responsecCtd)zAccess the status code from an HTTP response. Args: response (object): The HTTP response object. Raises: NotImplementedError: Always, since virtual. This implementation is virtual.NotImplementedErrorr*rrrr(h zUploadBase._get_status_codecCr,)zAccess the headers from an HTTP response. Args: response (object): The HTTP response object. Raises: NotImplementedError: Always, since virtual. r-r.r0rrr _get_headerstr1zUploadBase._get_headerscCr,)zAccess the response body from an HTTP response. Args: response (object): The HTTP response object. Raises: NotImplementedError: Always, since virtual. r-r.r0rrr _get_bodyr1zUploadBase._get_bodyN) __name__ __module__ __qualname____doc__rpropertyr"r+ staticmethodr(r2r3rrrrr8s     rc@s"eZdZdZddZdddZdS) SimpleUploadaUpload a resource to a Google API. A **simple** media upload sends no metadata and completes the upload in a single request. Args: upload_url (str): The URL where the content will be uploaded. headers (Optional[Mapping[str, str]]): Extra headers that should be sent with the request, e.g. headers for encrypted data. Attributes: upload_url (str): The URL where the content will be uploaded. cCs@|jrtdt|tstdt|||jt<t|j ||jfS)aPrepare the contents of an HTTP request. This is everything that must be done before a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. .. note: This method will be used only once, so ``headers`` will be mutated by having a new key added to it. Args: data (bytes): The resource content to be uploaded. content_type (str): The content type for the request. Returns: Tuple[str, str, bytes, Mapping[str, str]]: The quadruple * HTTP verb for the request (always POST) * the URL for the request * the body of the request * headers for the request Raises: ValueError: If the current upload has already finished. TypeError: If ``data`` isn't bytes. .. _sans-I/O: https://sans-io.readthedocs.io/ An upload can only be used once.`data` must be bytes, received) r" ValueError isinstancebytes TypeErrortyperrrr)rdata content_typerrr_prepare_requests   zSimpleUpload._prepare_requestNcCr,)a Transmit the resource to be uploaded. Args: transport (object): An object which can make authenticated requests. data (bytes): The resource content to be uploaded. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. timeout (Optional[Union[float, aiohttp.ClientTimeout]]): The number of seconds to wait for the server response. Depending on the retry strategy, a request may be repeated several times using the same timeout each time. Can also be passed as an `aiohttp.ClientTimeout` object. Raises: NotImplementedError: Always, since virtual. r-r.)r transportrCrDtimeoutrrrtransmitszSimpleUpload.transmitr4)r5r6r7r8rErHrrrrr;s&r;cs4eZdZdZd fdd ZddZd ddZZS) MultipartUploada3Upload a resource with metadata to a Google API. A **multipart** upload sends both metadata and the resource in a single (multipart) request. Args: upload_url (str): The URL where the content will be uploaded. headers (Optional[Mapping[str, str]]): Extra headers that should be sent with the request, e.g. headers for encrypted data. checksum Optional([str]): The type of checksum to compute to verify the integrity of the object. The request metadata will be amended to include the computed value. Using this option will override a manually-set checksum value. Supported values are "md5", "crc32c" and None. The default is None. Attributes: upload_url (str): The URL where the content will be uploaded. Ncstt|j||d||_dS)Nr)superrIr_checksum_type)rrrchecksum __class__rrrs zMultipartUpload.__init__c Cs|jrtdt|tstdt|t|j}|dur3| |t | }t |j}|||<t |||\}}t|d} | |jt<t|j||jfS)ajPrepare the contents of an HTTP request. This is everything that must be done before a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. .. note: This method will be used only once, so ``headers`` will be mutated by having a new key added to it. Args: data (bytes): The resource content to be uploaded. metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. Returns: Tuple[str, str, bytes, Mapping[str, str]]: The quadruple * HTTP verb for the request (always POST) * the URL for the request * the body of the request * headers for the request Raises: ValueError: If the current upload has already finished. TypeError: If ``data`` isn't bytes. .. _sans-I/O: https://sans-io.readthedocs.io/ r<r=N")r"r>r?r@rArB sync_helpers_get_checksum_objectrLupdateprepare_checksum_digestdigest_get_metadata_keyconstruct_multipart_requestrrrrr) rrCmetadatarDchecksum_objectactual_checksum metadata_keycontentmultipart_boundarymultipart_content_typerrrrEs$!      z MultipartUpload._prepare_requestcCr,)aqTransmit the resource to be uploaded. Args: transport (object): An object which can make authenticated requests. data (bytes): The resource content to be uploaded. metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. timeout (Optional[Union[float, aiohttp.ClientTimeout]]): The number of seconds to wait for the server response. Depending on the retry strategy, a request may be repeated several times using the same timeout each time. Can also be passed as an `aiohttp.ClientTimeout` object. Raises: NotImplementedError: Always, since virtual. r-r.)rrFrCrXrDrGrrrrH)zMultipartUpload.transmitNNr4)r5r6r7r8rrErH __classcell__rrrNrrIs :rIcseZdZdZd&fdd ZeddZeddZed d Zed d Z ed dZ d'ddZ ddZ   d(ddZ ddZddZddZddZd)ddZd d!Zd"d#Zd$d%ZZS)*ResumableUploadaInitiate and fulfill a resumable upload to a Google API. A **resumable** upload sends an initial request with the resource metadata and then gets assigned an upload ID / upload URL to send bytes to. Using the upload URL, the upload is then done in chunks (determined by the user) until all bytes have been uploaded. Args: upload_url (str): The URL where the resumable upload will be initiated. chunk_size (int): The size of each chunk used to upload the resource. headers (Optional[Mapping[str, str]]): Extra headers that should be sent with the :meth:`initiate` request, e.g. headers for encrypted data. These **will not** be sent with :meth:`transmit_next_chunk` or :meth:`recover` requests. checksum Optional([str]): The type of checksum to compute to verify the integrity of the object. After the upload is complete, the server-computed checksum of the resulting object will be read and google.resumable_media.common.DataCorruption will be raised on a mismatch. The corrupted file will not be deleted from the remote host automatically. Supported values are "md5", "crc32c" and None. The default is None. Attributes: upload_url (str): The URL where the content will be uploaded. Raises: ValueError: If ``chunk_size`` is not a multiple of :data:`.UPLOAD_CHUNK_SIZE`. Ncsvtt|j||d|tjdkrtdtjd||_d|_d|_ d|_ d|_ ||_ d|_ d|_d|_d|_dS)NrJrz{} KB must divide chunk sizeiF)rKrbrrUPLOAD_CHUNK_SIZEr>format _chunk_size_stream _content_type_bytes_uploaded_bytes_checksummedrL_checksum_object _total_bytes_resumable_url_invalid)rr chunk_sizerMrrNrrr_s" zResumableUpload.__init__cCr )zbool: Indicates if the upload is in an invalid state. This will occur if a call to :meth:`transmit_next_chunk` fails. To recover from such a failure, call :meth:`recover`. rmr!rrrinvalidrszResumableUpload.invalidcCr )z8int: The size of each chunk used to upload the resource.)rer!rrrrn{r#zResumableUpload.chunk_sizecCr )z;Optional[str]: The URL of the in-progress resumable upload.)rlr!rrr resumable_urlr#zResumableUpload.resumable_urlcCr )z-int: Number of bytes that have been uploaded.)rhr!rrrbytes_uploadedr#zResumableUpload.bytes_uploadedcCr )aOptional[int]: The total number of bytes to be uploaded. If this upload is initiated (via :meth:`initiate`) with ``stream_final=True``, this value will be populated based on the size of the ``stream`` being uploaded. (By default ``stream_final=True``.) If this upload is initiated with ``stream_final=False``, :attr:`total_bytes` will be :data:`None` since it cannot be determined from the stream. )rkr!rrr total_bytess zResumableUpload.total_bytesTc Cs|jdur td|dkrtd||_||_tdd|i}|dur'||_n|r.t||_|jdur=d|j}||d<| |j t | d }t|j||fS) aPrepare the contents of HTTP request to initiate upload. This is everything that must be done before a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. Args: stream (IO[bytes]): The stream (i.e. file-like object) that will be uploaded. The stream **must** be at the beginning (i.e. ``stream.tell() == 0``). metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. total_bytes (Optional[int]): The total number of bytes to be uploaded. If specified, the upload size **will not** be determined from the stream (even if ``stream_final=True``). stream_final (Optional[bool]): Indicates if the ``stream`` is "final" (i.e. no more bytes will be added to it). In this case we determine the upload size from the size of the stream. If ``total_bytes`` is passed, this argument will be ignored. Returns: Tuple[str, str, bytes, Mapping[str, str]]: The quadruple * HTTP verb for the request (always POST) * the URL for the request * the body of the request * headers for the request Raises: ValueError: If the current upload has already been initiated. ValueError: If ``stream`` is not at the beginning. .. _sans-I/O: https://sans-io.readthedocs.io/ Nz'This upload has already been initiated.rzStream must be at beginning.zapplication/json; charset=UTF-8zx-upload-content-typez{:d}zx-upload-content-lengthutf-8)rqr>tellrfrgrrkget_total_bytesrdrSrjsondumpsencoderr) rstreamrXrDrs stream_finalrcontent_lengthpayloadrrr_prepare_initiate_requests& '     z)ResumableUpload._prepare_initiate_requestcCs2tj|tjjf|j|jdt|d|j|_ dS)aVProcess the response from an HTTP request that initiated upload. This is everything that must be done after a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. This method takes the URL from the ``Location`` header and stores it for future use. Within that URL, we assume the ``upload_id`` query parameter has been included, but we do not check. Args: response (object): The HTTP response object (need headers). .. _sans-I/O: https://sans-io.readthedocs.io/ callbacklocationN) rr$r%r&r'r( _make_invalidheader_requiredr2rlr)rrr_process_initiate_responses z*ResumableUpload._process_initiate_responsecCr,)ahInitiate a resumable upload. By default, this method assumes your ``stream`` is in a "final" state ready to transmit. However, ``stream_final=False`` can be used to indicate that the size of the resource is not known. This can happen if bytes are being dynamically fed into ``stream``, e.g. if the stream is attached to application logs. If ``stream_final=False`` is used, :attr:`chunk_size` bytes will be read from the stream every time :meth:`transmit_next_chunk` is called. If one of those reads produces strictly fewer bites than the chunk size, the upload will be concluded. Args: transport (object): An object which can make authenticated requests. stream (IO[bytes]): The stream (i.e. file-like object) that will be uploaded. The stream **must** be at the beginning (i.e. ``stream.tell() == 0``). metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. total_bytes (Optional[int]): The total number of bytes to be uploaded. If specified, the upload size **will not** be determined from the stream (even if ``stream_final=True``). stream_final (Optional[bool]): Indicates if the ``stream`` is "final" (i.e. no more bytes will be added to it). In this case we determine the upload size from the size of the stream. If ``total_bytes`` is passed, this argument will be ignored. timeout (Optional[Union[float, aiohttp.ClientTimeout]]): The number of seconds to wait for the server response. Depending on the retry strategy, a request may be repeated several times using the same timeout each time. Can also be passed as an `aiohttp.ClientTimeout` object. Raises: NotImplementedError: Always, since virtual. r-r.)rrFrzrXrDrsr{rGrrrinitiates1zResumableUpload.initiatecCs|jrtd|jrtd|jdurtdt|j|j|j\}}}||jkr3t ||j}t|| ||t |j tj|i}t|j||fS)aPrepare the contents of HTTP request to upload a chunk. This is everything that must be done before a request that doesn't require network I/O. This is based on the `sans-I/O`_ philosophy. For the time being, this **does require** some form of I/O to read a chunk from ``stream`` (via :func:`get_next_chunk`). However, this will (almost) certainly not be network I/O. Returns: Tuple[str, str, bytes, Mapping[str, str]]: The quadruple * HTTP verb for the request (always PUT) * the URL for the request * the body of the request * headers for the request The headers **do not** incorporate the ``_headers`` on the current instance. Raises: ValueError: If the current upload has finished. ValueError: If the current upload is in an invalid state. ValueError: If the current upload has not been initiated. ValueError: If the location in the stream (i.e. ``stream.tell()``) does not agree with ``bytes_uploaded``. .. _sans-I/O: https://sans-io.readthedocs.io/ zUpload has finished.z;Upload is in an invalid state. To recover call `recover()`.Nz_This upload has not been initiated. Please call initiate() before beginning to transmit chunks.)r"r>rprqget_next_chunkrfrerkrrrrd_update_checksumrrgrCONTENT_RANGE_HEADERr)r start_byter} content_rangemsgrrrrrE%s*    z ResumableUpload._prepare_requestcCs d|_dS)zSimple setter for ``invalid``. This is intended to be passed along as a callback to helpers that raise an exception so they can mark this instance as invalid before raising. TNror!rrrr^s zResumableUpload._make_invalidcstj|tjjtjjf|j|jd}|tjjkr+|j||_d|_ | |IdHdStj |tj |j |jd}t|}|durL|t|d|dt|dd|_dS)aProcess the response from an HTTP request. This is everything that must be done after a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. Args: response (object): The HTTP response object. bytes_sent (int): The number of bytes sent in the request that ``response`` was returned for. Raises: ~google.resumable_media.common.InvalidResponse: If the status code is 308 and the ``range`` header is not of the form ``bytes 0-{end}``. ~google.resumable_media.common.InvalidResponse: If the status code is not 200 or 308. .. _sans-I/O: https://sans-io.readthedocs.io/ rTNUnexpected "range" header*Expected to be of the form "bytes=0-{end}"end_byte)rr$r%r&r'PERMANENT_REDIRECTr(rrhr_validate_checksumr RANGE_HEADERr2rmatchrInvalidResponseintgroup)rr* bytes_sent status_code bytes_rangerrrr_process_resumable_responsegs6   z+ResumableUpload._process_resumable_responsecs|jdurdSt|j}|IdH}||}|dur+t|t|| |t |j }||krEt |t|j||dS)aQCheck the computed checksum, if any, against the response headers. Args: response (object): The HTTP response object. Raises: ~google.resumable_media.common.DataCorruption: If the checksum computed locally and the checksum reported by the remote host do not match. N)rLrQrVrwgetrrrrdr2rTrjrUDataCorruptionrupper)rr*r[rXremote_checksumlocal_checksumrrrrs.    z"ResumableUpload._validate_checksumcCr,)aTransmit the next chunk of the resource to be uploaded. If the current upload was initiated with ``stream_final=False``, this method will dynamically determine if the upload has completed. The upload will be considered complete if the stream produces fewer than :attr:`chunk_size` bytes when a chunk is read from it. Args: transport (object): An object which can make authenticated requests. timeout (Optional[Union[float, aiohttp.ClientTimeout]]): The number of seconds to wait for the server response. Depending on the retry strategy, a request may be repeated several times using the same timeout each time. Can also be passed as an `aiohttp.ClientTimeout` object. Raises: NotImplementedError: Always, since virtual. r-r.)rrFrGrrrtransmit_next_chunkr_z#ResumableUpload.transmit_next_chunkcCs&|jstdtjdi}t|jd|fS)aPrepare the contents of HTTP request to recover from failure. This is everything that must be done before a request that doesn't require network I/O. This is based on the `sans-I/O`_ philosophy. We assume that the :attr:`resumable_url` is set (i.e. the only way the upload can end up :attr:`invalid` is if it has been initiated. Returns: Tuple[str, str, NoneType, Mapping[str, str]]: The quadruple * HTTP verb for the request (always PUT) * the URL for the request * the body of the request (always :data:`None`) * headers for the request The headers **do not** incorporate the ``_headers`` on the current instance. Raises: ValueError: If the current upload is not in an invalid state. .. _sans-I/O: https://sans-io.readthedocs.io/ z3Upload is not in invalid state, no need to recover.z bytes */*N)rpr>rrrrq)rrrrr_prepare_recover_requests z(ResumableUpload._prepare_recover_requestcCst|tjjf|j||}tj|vr6|tj}t |}|dur+t |d|dt | dd|_nd|_|j|jd|_dS)aProcess the response from an HTTP request to recover from failure. This is everything that must be done after a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. Args: response (object): The HTTP response object. Raises: ~google.resumable_media.common.InvalidResponse: If the status code is not 308. ~google.resumable_media.common.InvalidResponse: If the status code is 308 and the ``range`` header is not of the form ``bytes 0-{end}``. .. _sans-I/O: https://sans-io.readthedocs.io/ NrrrrrF)rr$r%r&rr(r2rrrrrrrrhrfseekrm)rr*rrrrrr_process_recover_responses(     z)ResumableUpload._process_recover_responsecCr,)a!Recover from a failure. This method should be used when a :class:`ResumableUpload` is in an :attr:`~ResumableUpload.invalid` state due to a request failure. This will verify the progress with the server and make sure the current upload is in a valid state before :meth:`transmit_next_chunk` can be used again. Args: transport (object): An object which can make authenticated requests. Raises: NotImplementedError: Always, since virtual. r-r.)rrFrrrrecover!szResumableUpload.recoverr`)NT)NTNr4)r5r6r7r8rr9rprnrqrrrsr~rrrErrrrrrrrarrrNrrb@s8      @  39 ; +rbcCs ttj}t|}|dS)zGet a random boundary for a multipart request. Returns: bytes: The boundary used to separate parts of a multipart request. rt)random randrangesysmaxsizer rdry) random_intboundaryrrr get_boundary5s   rcCsht}t|d}|d}t|}|t|t|td|tt|t|t}||fS)aConstruct a multipart request body. Args: data (bytes): The resource content (UTF-8 encoded as bytes) to be uploaded. metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. Returns: Tuple[bytes, bytes]: The multipart request body and the boundary used between each part. rtscontent-type: )rrwrxryr r r )rCrXrDr] json_bytes boundary_sepr\rrrrWBsB      rWcCs,|}|dtj|}|||S)zDetermine the total number of bytes in a stream. Args: stream (IO[bytes]): The stream (i.e. file-like object). Returns: int: The number of bytes. r)rurosSEEK_END)rzcurrent_position end_positionrrrrvks   rvcCs|}|dur|||krdkrnn|||}n||}|d}t|}|dur:||kr9|d}n|dkrG|dkrFtdn|dkrOtdt|||}|||fS)aGet a chunk from an I/O stream. The ``stream`` may have fewer bytes remaining than ``chunk_size`` so it may not always be the case that ``end_byte == start_byte + chunk_size - 1``. Args: stream (IO[bytes]): The stream (i.e. file-like object). chunk_size (int): The size of the chunk to be read from the ``stream``. total_bytes (Optional[int]): The (expected) total number of bytes in the ``stream``. Returns: Tuple[int, bytes, str]: Triple of: * the start byte index * the content in between the start and end bytes (inclusive) * content range header for the chunk (slice) that has been read Raises: ValueError: If ``total_bytes == 0`` but ``stream.read()`` yields non-empty content. ValueError: If there is no data left to consume. This corresponds exactly to the case ``end_byte < start_byte``, which can only occur if ``end_byte == start_byte - 1``. Nrrz:Stream specified as empty, but produced non-empty content.z;Stream is already exhausted. There is no content remaining.)rureadlenr>get_content_range)rzrnrsrr}rnum_bytes_readrrrrrs,$    rcCs4|dur t||S||krt|St|||S)aConvert start, end and total into content range header. If ``total_bytes`` is not known, uses "bytes {start}-{end}/*". If we are dealing with an empty range (i.e. ``end_byte < start_byte``) then "bytes */{total}" is used. This function **ASSUMES** that if the size is not known, the caller will not also pass an empty range. Args: start_byte (int): The start (inclusive) of the byte range. end_byte (int): The end (inclusive) of the byte range. total_bytes (Optional[int]): The number of bytes in the byte range (if known). Returns: str: The content range header. N)rrdr r)rrrsrrrrs   r)*r8 http.clientr%rwrrrgooglergoogle._async_resumable_mediargoogle.resumable_mediarQr sync_uploadrgoogle.resumable_media._uploadrrrr r r r r rrrrrrrobjectrr;rIrbrrWrvrrrrrrs0      DUJix ) 9