0dZddlmZddlmZddlmZddlmZddlmZddlZddl Z ddl Z ddl Z ddl Z ddl mZdd lmZmZdd lmZed d ZGd deZdZdZdZddZ ddZ ddZy)zAHelper functions for composite upload tracker file functionality.)absolute_import)print_function)division)unicode_literals) namedtupleN)CommandException)WriteJsonDataToTrackerFile#RaiseUnwritableTrackerFileException)UTF8ObjectFromTrackerzobject_name generationc$eZdZdZdZdZdZdZdZy)_CompositeUploadTrackerEntryz7Enum class for composite upload tracker file JSON keys. componentscomponent_namecomponent_generationencryption_key_sha256prefixN) __name__ __module__ __qualname____doc__COMPONENTS_LISTCOMPONENT_NAMECOMPONENT_GENERATION ENC_SHA256PREFIX.platform/gsutil/gslib/parallel_tracker_file.pyrr&s? /#./&* &rrcd}d}g}d} t|d}|j}tj|}|tj }|tj }|tjD]?}|jt|tj|tjA |r|j) |||fS#t$rD} | jtjk7r|jd|| j Yd} ~ `d} ~ wt"t$f$r} d}t'\}}Yd} ~ d} ~ wwxYw#|r|j)wwxYw)aRead the tracker file from the last parallel composite upload attempt. If it exists, the tracker file is of the format described in WriteParallelUploadTrackerFile or a legacy format. If the file doesn't exist or is formatted incorrectly, then the upload will start from the beginning. This function is not thread-safe and must be protected by a lock if called within Command.Apply. Args: tracker_file_name: The name of the tracker file to read parse. logger: logging.Logger for outputting log messages. Returns: enc_key_sha256: Encryption key SHA256 used to encrypt the existing components, or None if an encryption key was not used. component_prefix: String prefix used in naming the existing components, or None if no prefix was found. existing_components: A list of ObjectFromTracker objects representing the set of files that have already been uploaded. Nrz^Couldn't read upload tracker file (%s): %s. Restarting parallel composite upload from scratch.)openreadjsonloadsrrrrappendr rrIOErrorerrnoENOENTwarnstrerrorKeyError ValueError_ParseLegacyTrackerDataclose) tracker_file_nameloggerenc_key_sha256rexisting_components tracker_file tracker_data tracker_json componentes rReadParallelUploadTrackerFiler9/sS,. &,)3/L$$&L::l+L!">"I"IJN 6== >F!">"N"NO   4CCD4IIJ LMP& &"5 66#  ww%,, kk 45F ** J JN$;L$I!V J sBB,C6E E:DEE)D>9E>EEEc|jddd}d}g}|r[|d}d}|t|dz krC||||dz}}|sd}|jt|||dz }|t|dz krC||fS)atParses a legacy parallel composite upload tracker file. Args: tracker_data: Legacy tracker file contents. Returns: component_prefix: The prefix used in naming the existing components, or None if no prefix was found. existing_components: A list of ObjectFromTracker objects representing the set of files that have already been uploaded.  Nr)splitlenr&r )r5old_tracker_datarr3iname generations rr.r.ks"''-cr2 & a F A c"#a' ',Q/1A!a%1HZt    !24!DE1fa c"#a' ' % &&rc tjrJt|tr|j t }t|tr|j t }|r||k7r |j d|g} |D]4} |j} | j| _| j| _ 6|j|| | tjj|jjdgfS||fS#|D cgc]} | jncc} w} } |j ddj!| YdgfSxYw)aValidates that tracker data matches the current encryption key. If the data does not match, makes a best-effort attempt to delete existing temporary component objects encrypted with the old key. Args: tracker_file_name: String file name of tracker file. existing_enc_sha256: Encryption key SHA256 used to encrypt the existing components, or None if an encryption key was not used. existing_prefix: String prefix used in naming the existing components, or None if no prefix was found. existing_components: A list of ObjectFromTracker objects representing the set of files that have already been uploaded. current_enc_key_sha256: Current Encryption key SHA256 that should be used to encrypt objects. bucket_url: Bucket URL in which the components exist. command_obj: Command class for calls to Apply. logger: logging.Logger for outputting log messages. delete_func: command.Apply-callable function for deleting objects. delete_exc_handler: Exception handler for delete_func. Returns: prefix: existing_prefix, or None if the encryption key did not match. existing_components: existing_components, or empty list if the encryption key did not match. zUpload tracker file (%s) does not match current encryption key. Deleting old components and restarting upload from scratch with a new tracker file that uses the current encryption key.) arg_checkerparallel_operations_overridezrFailed to delete some of the following temporary objects: %s (Continuing on to re-upload components from scratch.)r;N)sixPY3 isinstancestrencoder r*Clone object_namerDApplygslibcommandDummyArgCheckerParallelOverrideReasonSPEEDjoin)r0existing_enc_sha256existing_prefixr3current_enc_key_sha256 bucket_url command_objr1 delete_funcdelete_exc_handlercomponents_to_deleter7urlcomponent_namess r$ValidateParallelCompositeTrackerDatar`sR> WW%s+/66t<(#.5<rzsH&%'" ,E&24LM697x'>F0R P&EI PJ:>,Cr