`|dZddlmZddlmZddlmZddlmZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZdZdZdZdZGddeZdZ dZ!dZ" d&dZ#dZ$ d'dZ%dZ&dZ' d(dZ(dZ)dZ* d'dZ+ d'd Z,d!Z-d"Z.d#Z/d'd$Z0d%Z1y))z0Helper functions for tracker file functionality.)absolute_import)print_function)division)unicode_literalsN)config)CommandException)GetGsutilStateDir)ResumableThreshold)UTF8)GetMd5)CreateDirIfNeededdzCouldn't write tracker file (%s): %s. This can happen if gsutil is configured to save tracker files to an unwritable directory)encryption_key_sha256serialization_datac$eZdZdZdZdZdZdZdZy)TrackerFileTypeuploaddownloaddownload_componentparallel_uploadsliced_downloadrewriteN) __name__ __module__ __qualname__UPLOADDOWNLOADDOWNLOAD_COMPONENTPARALLEL_UPLOADSLICED_DOWNLOADREWRITE%platform/gsutil/gslib/tracker_file.pyrr4s! & (+%/%/ 'r#rct|tjr|jt}|}n|}|j t}t j|}d|jzdz|ddzS)a"Apply a hash function (SHA1) to shorten the passed file name. The spec for the hashed file name is as follows: TRACKER__ where hash is a SHA1 hash on the original file name and trailing is the last 16 chars from the original file name. Max file name lengths vary by operating system so the goal of this function is to ensure the hashed version takes fewer than 100 characters. Args: filename: file name to be hashed. May be unicode or bytes. Returns: shorter, hashed version of passed file name TRACKER_.iN) isinstancesix text_typeencoder decodehashlibsha1 hexdigest)filenamefilename_bytes filename_strms r$ _HashFilenamer4=sh$#--(__T*NLN??4(L ll>"! akkm #c )L,> >>r#ctjddtjj t d}t ||S)aLooks up or creates the gsutil tracker file directory. This is the configured directory where gsutil keeps its resumable transfer tracker files. This function creates it if it doesn't already exist. Returns: The pathname to the tracker directory. GSUtilresumable_tracker_dirz tracker-files)rgetospathjoinr r ) tracker_dirs r$CreateTrackerDirIfNeededr=Ys< 8%<77<<(9(;_MO+K r#ctjddd|d|d|d|d|d }t|tjS)aUGets the tracker file name described by the arguments. Args: src_bucket_name: Source bucket (string). src_obj_name: Source object (string). dst_bucket_name: Destination bucket (string). dst_obj_name: Destination object (string) api_selector: API to use for this operation. Returns: File path to tracker file. [/\\]_ rewrite____z.token)resub_HashAndReturnPathrr!)src_bucket_name src_obj_namedst_bucket_name dst_obj_name api_selectorres_tracker_file_names r$GetRewriteTrackerFilePathrLhs? 66 o| 1?3J3J KKr#c |tjk(r7tjddd|jd|j d|d}nt|tj k(rGtjdddtjj|j d|d}n|tjk(rEtjdddtjj|j ||fz}n|tjk(r9tjddd |jd|j d|d|d }nv|tjk(rFtjddd tjj|j d|d}n|tjk(r tt|S) aGets the tracker file name described by the arguments. Args: dst_url: Destination URL for tracker file. tracker_file_type: TrackerFileType for this operation. api_selector: API to use for this operation. src_url: Source URL for the source file name for parallel uploads. component_num: Component number if this is a download component, else None. Returns: File path to tracker file. r?r@resumable_upload__rBz.urlresumable_download__z.etagz#resumable_download__%s__%s__%d.etagparallel_upload__sliced_download__)rrrCrD bucket_name object_namerr9r:realpathrrr r!NotImplementedErrorrE)dst_urltracker_file_typerJsrc_url component_numrKs r$GetTrackerFilePathrZs"/000VV3   g11<ABO444VV3   '-- . >?O>>> VV3=   '-- . mLMNO;;; VV3   g117LJKO;;;VV3   '-- . >?O333   13D EEr#ctt|tj|t ||}|D] }t|y)zDeletes all tracker files corresponding to an object download. Args: dst_url: StorageUrl describing the destination file. api_selector: The Cloud API implementation used. N)DeleteTrackerFilerZrr!GetSlicedDownloadTrackerFilePaths)rVrJ tracker_files tracker_files r$DeleteDownloadTrackerFilesr`s?/":":LIK4G\J-#ll#$r#c t|tj|}|g}|;d} t|d}t j |d} |r|j t|D].}|jt|tj||0|S#t tf$r|cY|r|jSSwxYw#|r|jwwxYw)aGets a list of sliced download tracker file paths. The list consists of the parent tracker file path in index 0, and then any existing component tracker files in [1:]. Args: dst_url: Destination URL for tracker file. api_selector: API to use for this operation. num_components: The number of component tracker files, if already known. If not known, the number will be retrieved from the parent tracker file on disk. Returns: File path to tracker file. Nrnum_componentsrY) rZrr openjsonloadIOError ValueErrorcloserangeappendr)rVrJrcparallel_tracker_file_pathtracker_file_pathsr_is r$r]r]s" 2 ..  >23L4c:lyy./?@n   a7*==')* ,- !  Z       s#$BC+CCCCct}t|}t|jd|}|tj |}t |tksJ|S)zHashes and returns a tracker file path. Args: res_tracker_file_name: The tracker file name prior to it being hashed. tracker_file_type: The TrackerFileType of res_tracker_file_name. Returns: Final (hashed) tracker file path. r@)r=r4strlowerr9seplenMAX_TRACKER_FILE_NAME_LENGTH)rKrWr7hashed_tracker_file_nametracker_file_nametracker_file_paths r$rErEse34*+@A!$%6!7!=!=!?!9;"7"35  "> >> > r#ct|r6tjj|rtj|yyyN)r9r:existsunlink)rws r$r\r\s*277>>*;<II =r#c L|r@|jr4|jr(|jr|r|jr|jr|syt} |||||||| ||| f D]8} | | j t j | jd:| jS)a`Creates an MD5 hex digest of the parameters for a rewrite call. Resuming rewrites requires that the input parameters are identical. Thus, the rewrite tracker file needs to represent the input parameters. For easy comparison, hash the input values. If a user does a performs a same-source/same-destination rewrite via a different command (for example, with a changed ACL), the hashes will not match and we will restart the rewrite from the beginning. Args: src_obj_metadata: apitools Object describing source object. Must include bucket, name, and etag. dst_obj_metadata: apitools Object describing destination object. Must include bucket and object name projection: Projection used for the API call. src_generation: Optional source generation. gen_match: Optional generation precondition. meta_gen_match: Optional metageneration precondition. canned_acl: Optional canned ACL string. max_bytes_per_call: Optional maximum bytes rewritten per call. src_dec_key_sha256: Optional SHA256 hash string of decryption key for source object. dst_enc_key_sha256: Optional SHA256 hash string of encryption key for destination object. fields: Optional fields to include in response to call. Returns: MD5 hex digest Hash of the input parameters, or None if required parameters are missing. Nr ) bucketnameetagr updater)r*r+r/) src_obj_metadatadst_obj_metadata projectionsrc_generation gen_matchmeta_gen_match canned_aclmax_bytes_per_callsrc_dec_key_sha256dst_enc_key_sha256fieldsmd5_hash input_params r$HashRewriteParametersrsR "2"9"9   '7'<'< "2"9"9   z X(  koocmmK077?@! "    r#cd}|sy t|d}|jjd}||k(r2|jjd|r|jSS |r|jyy#t$rW}|j t j k7r0tjjd|d|jdYd}~od}~wwxYw#|r|jwwxYw)a~Attempts to read a rewrite tracker file. Args: tracker_file_name: Tracker file path string. rewrite_params_hash: MD5 hex digest of rewrite call parameters constructed by HashRewriteParameters. Returns: String rewrite_token for resuming rewrite requests if a matching tracker file exists, None otherwise (which will result in starting a new rewrite). Nrb z!Couldn't read Copy tracker file (): z. Restarting copy from scratch.) rereadlinerstriprjrherrnoENOENTsysstderrwritestrerror)rwrewrite_params_hashr_ existing_hashes r$ReadRewriteTrackerFilerKs,  )3/L ))+2248M++  " " $ + +D 1, ?ww%,, jj/ =??s1AA>)C!> CA CC!CC!!C6c(t||d|dy)a Writes a rewrite tracker file. Args: tracker_file_name: Tracker file path string. rewrite_params_hash: MD5 hex digest of rewrite call parameters constructed by HashRewriteParameters. rewrite_token: Rewrite token string returned by the service. rN)_WriteTrackerFile)rwr rewrite_tokens r$WriteRewriteTrackerFilerls%"5}EGr#c |jsJd}|jtkr||fS|j}|tj } ntj } |d|zz }t|| ||}d} t|d} | tj urE| jjd} | |jk(r||f| r| jSS| tj uratj| j} | d|jk(r,| d|jk(r|| df| r| jSS|j!d |z| r| j | tj urt/|d |jz||fS| tj ur t1|||||fS#t"t$f$rX} t'| t$s| j(t(j*k7r!|j!d |d t-| d Yd} ~ d} ~ wwxYw#| r| jwwxYw)a?Checks for a download tracker file and creates one if it does not exist. The methodology for determining the download start point differs between normal and sliced downloads. For normal downloads, the existing bytes in the file are presumed to be correct and have been previously downloaded from the server (if a tracker file exists). In this case, the existing file size is used to determine the download start point. For sliced downloads, the number of bytes previously retrieved from the server cannot be determined from the existing file size, and so the number of bytes known to have been previously downloaded is retrieved from the tracker file. Args: src_obj_metadata: Metadata for the source object. Must include etag and generation. dst_url: Destination URL for tracker file. logger: For outputting log messages. api_selector: API to use for this operation. start_byte: The start byte of the byte range for this download. existing_file_size: Size of existing file for this download on disk. component_num: The component number, if this is a component of a parallel download, else None. Returns: tracker_file_name: The name of the tracker file, if one was used. download_start_byte: The first byte that still needs to be downloaded. Nz component %drdrbrr generationdownload_start_bytezPTracker file doesn't match for download of %s. Restarting download from scratch.z%Couldn't read download tracker file (rz#. Restarting download from scratch.z%s )rsizer rSrrrrZrerrrjrfloadsreadrwarnrhrir(rrrqr!WriteDownloadComponentTrackerFile)rrVloggerrJ start_byteexisting_file_sizerYrw download_namerWr_ etag_valuecomponent_datars r$ReadOrCreateDownloadTrackerFilerzs]B    /11 j ((%%-'00'::_}44M():)57DF,)3/LO444((*11$7j ',, , "44"! o@@ @zz,"3"3"56n  $4$9$9 9  &*:*E*E E .1F"GG KK)+89:/222'2B2G2G)GH J &&O>>>%&79I&02 J &&# : J!Z AGGu||$; kk.?QIJ Js95AGAG GH+AH&!H.&H++H..Ic|jsJd}|jtkr|S|tj}ntj }t ||||}d} t|d}|tjurC|jjd} | |jk(r||r|jSS|tj ur_tj|j} | d|jk(r*| d|jk(r| d|r|jSS|r|j|S#tt f$rY%wxYw#|r|jwwxYw)afReturns the download starting point. The methodology of this function is the same as in ReadOrCreateDownloadTrackerFile, with the difference that we are not interested here in possibly creating a tracker file. In case there is no tracker file, this means the download starting point is start_byte. Args: src_obj_metadata: Metadata for the source object. Must include etag and generation. dst_url: Destination URL for tracker file. api_selector: API to use for this operation. start_byte: The start byte of the byte range for this download. existing_file_size: Size of existing file for this download on disk. component_num: The component number, if this is a component of a parallel download, else None. Returns: download_start_byte: The first byte that still needs to be downloaded. Nrdrbrrrr)rrr rrrrZrerrrjrfrrrrhri) rrVrJrrrYrwrWr_rrs r$GetDownloadStartByters4    /11 '00'::():)57DF,)3/LO444((*11$7j ',, ,! o@@ @zz,"3"3"56n  $4$9$9 9  &*:*E*E E34  :   s,A EAEEEEEE/cv|j|j|d}t|tj|y)zUpdates or creates a download component tracker file on disk. Args: tracker_file_name: The name of the tracker file. src_obj_metadata: Metadata for the source object. Must include etag. current_file_pos: The current position in the file. )rrrN)rrrrfdumps)rwrcurrent_file_posrs r$rrs6##$//-. %tzz.'ABr#cf tj|tjtjztjzd}tj |d5}|j |dddy#1swYyxYw#ttf$r}t||jd}~wwxYw)z/Creates a tracker file, storing the input data.iwNF) r9reO_WRONLYO_CREATO_TRUNCfdopenrrhOSError#RaiseUnwritableTrackerFileExceptionr)rwdatafdtfrs r$rr+sM "BKK"**$d}d}d} t|d}|j}tj|}|t|k7rd}d}n=|t |r|j |r|jd||r t|SS |r|j |r|jd||r t|yy#t$rD}|jtjk7r|jd||jYd}~}d}~wttf$rL}d}|d}n;cYd}~|r|j |r|jd||r t|SSYd}~d}~wwxYw#|r|j |r|jd||r t|wwxYw)aReads tracker data from an upload tracker file if it exists. Deletes the tracker file if it uses an old format or the desired encryption key has changed. Args: tracker_file_name: Tracker file name for this upload. logger: logging.Logger for outputting log messages. encryption_key_sha256: Encryption key SHA256 for use in this upload, if any. Returns: Serialization data if the tracker file already exists (resume existing upload), None otherwise. NFrbTzUpload tracker file (%s) does not match current encryption key. Restarting upload from scratch with a new tracker file that uses the current encryption key.zKCouldn't read upload tracker file (%s): %s. Restarting upload from scratch.)rerrfrENCRYPTION_UPLOAD_TRACKER_ENTRY"SERIALIZATION_UPLOAD_TRACKER_ENTRYrjrr\rhrrrKeyErrorri) rwrrr_remove_tracker_fileencryption_restart tracker_data tracker_jsonrs r$GetUploadTrackerDatarDs,#+)3/L$$&L::l+L348MM  < =& kk 78IK)*9* kk 78IK)*3 A ww%,, kk !"3QZZA J ( kk 78IK)*  kk 78IK)*sNA B=E&= E#:DE&E#EE# E&E&E##E&&6Fc*tt||fz)z:Raises an exception when unable to write the tracker file.)r&TRACKER_FILE_UNWRITABLE_EXCEPTION_TEXT)rw error_strs r$rrs?+Y78 99r#)NNrz)NNNNNNNN)2__doc__ __future__rrrrrr-rfr9rCrr)botorgslib.exceptionrgslib.utils.boto_utilr r gslib.utils.constantsr gslib.utils.hashing_helperr gslib.utils.system_utilr rurrrobjectrr4r=rLrZr`r]rEr\rrrrrrrrrrr"r#r$rs7&%'  ,34&-5 #C' #:%9"f?8 L6 $%) 0Ff$&6:(V(!*.$()-%)-1-1-1!%@FB G(37 V'|(, CLC$ M 18+v9r#