$\dZddlmZddlmZddlmZddlZddlZddlZddlZddl m Z ddl m Z ddl m Z dd l mZdd l mZdd l mZdd l mZdd l mZddl mZddl mZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!dddZ"dZ#dZ$dZ% d dZ&dZ'y)!z2Utility functions for performing upload operation.)absolute_import)division)unicode_literalsN) api_factory) cloud_api)request_config_factory)buffered_upload_stream)component_stream)errors)fast_crc32c_util) hash_util)progress_callbacks) upload_stream) task_status) delete_task)crc32c) properties)files)hashing) platforms)scaled_integerz text/markdownzapplication/gzip)z.mdz.tgzcJtjtjjj j }||k\r@tjj|jvrtjjStjjS)a Determines if resumbale uplaod should be performed. Args: api (CloudApi): An api instance to check if it supports resumable upload. object_length (int): Length of the data to be uploaded. Returns: bool: True if resumable upload can be performed. )r ParseIntegerrVALUESstorageresumable_thresholdGetr CapabilityRESUMABLE_UPLOAD capabilitiesUploadStrategy RESUMABLESIMPLE)api object_lengthrs >lib/googlecloudsdk/command_lib/storage/tasks/cp/upload_util.pyget_upload_strategyr'5sz'3333779;**++s/?/??  # # - --  # # * **c |rtjStjD]\}}|j |s|cSt j jsxtjjjjrFtjddd|gdtjd}|j j#}nt%j&|\}}|r|StjS)a%Gets a file's MIME type. Favors returning the result of `file -b --mime ...` if the command is available and users have enabled it. Otherwise, it returns a type based on the file's extension. Args: source_path (str): Path to file. May differ from file_resource.storage_url if using a temporary file (e.g. for gzipping). is_stream (bool): If the source file is a pipe (typically FIFO or stdin). Returns: A MIME type (str). If a type cannot be guessed, request_config_factory.DEFAULT_CONTENT_TYPE is returned. filez-bz--mimeT)checkstdoutuniversal_newlines)rDEFAULT_CONTENT_TYPECOMMON_EXTENSION_RULESitemsendswithrOperatingSystem IsWindowsrrr use_magicfileGetBool subprocessrunPIPEr,strip mimetypes guess_type) source_path is_stream extension content_typeoutput_s r&get_content_typerBHs" ! 6 66"8!=!=!?iI& "@  # # - - /--557 ^^VT8[A"&#-??/35F==&&(L**;7OL!   4 44r(c|jj}tjjj j r|jjnd}tj||}tjtjjjj}tjj|vrut!j"|tjj$k(st!j&r.t(j*j,t!j.iSiS|j0s9tjj2|vs|tjj4k(riSt(j*j6t9j:iS)aiGets appropriate hash objects for upload validation. Args: source_resource (resource_reference.FileObjectResource): The upload source. destination_resource (resource_reference.ObjectResource): The upload destination. Returns: A dict[hash_util.HashAlgorithm, hash object], the values of which should be updated with uploaded bytes. N) storage_urlschemerrr#enable_zonal_buckets_bidi_streamingr5 bucket_namerget_capabilities CheckHashes check_hashesrrrAPPENDABLE_UPLOADr log_or_raise_crc32c_issuesALWAYScheck_if_will_use_fast_crc32cr HashAlgorithmCRC32C get_crc32cmd5_hashCLIENT_SIDE_HASH_VALIDATIONNEVERMD5rget_md5)source_resourcedestination_resourceproviderrGr rJs r& get_digestersrZpsL" - - 4 4(    " " F F N N P&&22  --h D,'',,0024,++|;//1 ..555  9 9 ;%%,,.>.I.I.K LL I66,Fj,,222 I  ! ! % %w'8 99r(c |rttj||xsd||j|j||tjj t jtj }nd}|jjrt jdd} n)tj|jj} |jjr^t!j"t$j&j(j*j-} t/j0| | ||S|t3j4| |||St7j8| ||||S)aGets a stream to use for an upload. Args: source_resource (resource_reference.FileObjectResource): Contains a path to the source file. length (int|None): The total number of bytes to be uploaded. offset (int|None): The position of the first byte to be uploaded. digesters (dict[hash_util.HashAlgorithm, hash object]|None): Hash objects to be populated as bytes are read. task_status_queue (multiprocessing.Queue|None): Used for sending progress messages. If None, no messages will be generated or sent. destination_resource (resource_reference.ObjectResource|None): The upload destination. Used for progress reports, and should be specified if task_status_queue is. component_number (int|None): Identifies a component in composite uploads. total_components (int|None): The total number of components used in a composite upload. Returns: An UploadStream wrapping the file specified by source_resource. r) status_queueoffsetlength source_urldestination_urlcomponent_numbertotal_componentsoperation_name process_id thread_idNrb)max_buffer_size digestersprogress_callback)r^rhri)r]r^rhri)rFilesAndBytesProgressCallbackrDr OperationName UPLOADINGosgetpid threading get_identis_stdiofdopenrBinaryFileReader resource_namer=rParseBinaryIntegerrrrupload_chunk_sizerr BufferedUploadStreamr UploadStreamr ComponentStream) rWr^r]rhtask_status_queuerXrarbri source_streamrgs r& get_streamr|sL:*HH&{"..,88))"00::99;%%'   ))IIa&M**##113M  **$77!!33779;O ! 6 6'+  -- ~  % %+  --  + +fVy+ --r(c|sytjj|vr=tj|tjj}|j}ntjj |vr=t j|tjj }|j}nAtjdjdj|j tj|jj ||y#tj"$r1t%j&|jj)|wxYw)zHRaises error if hashes for uploaded_resource and digesters do not match.Nz4Unsupported hash algorithm(s) found in digesters: {}z, )rz)r rOrUget_base64_hash_digest_stringrRrPrget_hash crc32c_hashr Errorformatjoinkeysvalidate_object_hashes_matchrD url_stringHashMismatchErrorrDeleteObjectTaskexecute)rhuploaded_resourcerzcalculated_digestdestination_hashs r&validate_uploaded_objectrs1    I-!??)))--.0(11%%2)))001)44 ,,>EE IIinn& '     **%%002C  ! !   !2!>!>?GG+H  s 8+D$$AE()NNNNNNN)(__doc__ __future__rrrr:rmr6rogooglecloudsdk.api_lib.storagerrr"googlecloudsdk.command_lib.storager r r r r rr(googlecloudsdk.command_lib.storage.tasksr+googlecloudsdk.command_lib.storage.tasks.rmrgooglecloudsdk.command_lib.utilrgooglecloudsdk.corergooglecloudsdk.core.utilrrrrr/r'rBrZr|rr(r&rs 9&' 64AE?5?8A<@C2**,.3   +&%5P$:P!%$( $ $D-N r(