"NdZddlmZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l m Z dd l mZdd l mZddlZd ZGd dej$ej&ZGddeZGddeZGddeZGddeZedfdZedfdZedfdZy)aUtilities for parallelizing Cloud Storage file operations. Example usage (for simplicity, use absolute *nix-style paths; in practice you'll want to use os.path.join and friends): >>> upload_tasks = [ ... FileUploadTask('/tmp/file1.txt', 'gs://my-bucket', ... 'path/to/remote1.txt'), ... FileUploadTask('/tmp/file2.txt', 'gs://my-bucket', '/remote2.txt') ... ] >>> UploadFiles(upload_tasks, num_threads=16) This will block until all files are uploaded, using 16 threads (but just the current process). Afterwards, there will be objects at 'gs://my-bucket/path/to/remote1.txt' and 'gs://my-bucket/remote2.txt'. >>> delete_tasks = [ ... ObjectDeleteTask('gs://my-bucket', 'path/to/remote1.txt'), ... ObjectDeleteTask('gs://my-bucket', '/remote2.txt') ... ] >>> DeleteObjects(delete_tasks, num_threads=16) This removes the objects uploaded in the last code snippet. )absolute_import)division)unicode_literalsN) storage_api)log) console_io)parallel)retry)textc6eZdZdZej ddZy)Taskz8Base clase for a storage tasks that can be parallelized.NcyN)selfcallbacks :lib/googlecloudsdk/command_lib/storage/storage_parallel.pyExecutez Task.Execute?sr)__name__ __module__ __qualname____doc__abcabstractmethodrrrrrr<s@  rrc0eZdZdZdZdZdZdZddZy) FileUploadTaskaSelf-contained representation of a file to upload and its destination. Attributes: source_local_path: str, The local filesystem path of the source file to upload. dest_obj_ref: storage_util.ObjectReference, The object the file will be copied to. c ||_||_yr)source_local_path dest_obj_ref)rr r!s r__init__zFileUploadTask.__init__Ns.D$Drcjdj|j|jjS)NzUpload: {} --> {}formatr r!ToUrlrs r__str__zFileUploadTask.__str__Rs/  % %  1 1 7 7 9 ;;rcldj|j|jjS)Nz@FileUploadTask(source_path={source_path}, dest_path={dest_path}) source_path dest_pathr$r's r__repr__zFileUploadTask.__repr__Vs3J D22++113  56rcDt|j|jfSr)hashr r!r's r__hash__zFileUploadTask.__hash__\s ''):):; < {}r%rBr&rCr's rr(zFileDownloadTask.__str__us/ ' ' !!#T%9%9 ;;rcldj|jj|jS)NzBFileDownloadTask(source_path={source_path}, dest_path={dest_path})r*rEr's rr-zFileDownloadTask.__repr__ys3L D//557..  01rcDt|j|jfSr)r/rBrCr's rr0zFileDownloadTask.__hash__s $$d&:&:; <rrrr@r@hs +;1 =rr@c0eZdZdZdZdZdZdZddZy) FileRemoteCopyTaskaSelf-contained representation of a copy between GCS objects. Attributes: source_obj_ref: storage_util.ObjectReference, The object reference of the file to download. dest_obj_ref: storage_util.ObjectReference, The object reference to write the file to. c ||_||_yr)rBr!)rrBr!s rr"zFileRemoteCopyTask.__init__s(D$Drcdj|jj|jjS)NzCopy: {} --> {}r%rBr&r!r's rr(zFileRemoteCopyTask.__str__s:  # # !!#T%6%6%<%<%> @@rcdj|jj|jjS)NzDFileRemoteCopyTask(source_path={source_path}, dest_path={dest_path})r*rNr's rr-zFileRemoteCopyTask.__repr__s<N D//557++113  56rcDt|j|jfSr)r/rBr!r's rr0zFileRemoteCopyTask.__hash__s $$d&7&78 99rNctj}tjdj |j |j |jf|r|yyr2)rr8r r9r:CopyrBr!r<s rrzFileRemoteCopyTask.ExecutesX ..0N MMq!22!!4#4#4 537jrrr>rrrrKrKs!%@6 :rrKc0eZdZdZdZdZdZdZddZy) ObjectDeleteTaskzSelf-contained representation of an object to delete. Attributes: obj_ref: storage_util.ObjectReference, The object to delete. c||_yr)obj_ref)rrVs rr"zObjectDeleteTask.__init__s DLrcTdj|jjS)Nz Delete: {}r%rVr&r's rr(zObjectDeleteTask.__str__s   t||113 44rcVdj|jjS)NzObjectDeleteTask(object={obj})objrXr's rr-zObjectDeleteTask.__repr__s# * 1 1dll6H6H6J 1 KKrc,t|jSr)r/rVr's rr0zObjectDeleteTask.__hash__s  rNctj}tjdj |j |j f|r|yy)z8Complete one ObjectDeleteTask (safe to run in parallel).r3r4r6N)rr8r r9r: DeleteObjectrVr<s rrzObjectDeleteTask.ExecutesJ ..0N MMq!22##4<rrrrTrTs! 5LrrTctj|tjd|tj|}|r,t j t ||}|jnt j}d|dk(r%|5|D]}|j dddy|5|5|jfd|ddddddy#1swYyxYw#1swYxYw#1swYyxYw)aPerform the given storage tasks in parallel. Factors out common work: logging, setting up parallelism, managing a progress bar (if necessary). Args: tasks: [Operation], To be executed in parallel. num_threads: int, The number of threads to use progress_bar_label: str, If set, a progress bar will be shown with this label. Otherwise, no progress bar is displayed. zUsing [%d] threadsNrc&|jSr)r)taskrs rzExecuteTasks..sDLL2r) rdebugr GetPoolrTickableProgressBarlenTickNoOpProgressBarrMap)tasks num_threadsprogress_bar_labelpool progress_bartrs @r ExecuteTasksros)) )) +.   + &$11 E &(L  H--/LHA ! (  t hh2E: ts0 C2C45C( C4C%(C1 -C44C=Fct|}|r'dj|tj|d}nd}t |||y)alUpload the given files to the given Cloud Storage URLs. Uses the appropriate parallelism (multi-process, multi-thread, both, or synchronous). Args: files_to_upload: list of FileUploadTask num_threads: int (optional), the number of threads to use. show_progress_bar: bool. If true, show a progress bar to the users when uploading files. z'Uploading {} {} to Google Cloud StoragefileNrer%r Pluralizero)files_to_uploadrjshow_progress_bar num_fileslabels r UploadFilesrxsD/") 5 < <4>>)V4 6E E U3rct|}|r'dj|tj|d}nd}t |||y)a_Delete the given Cloud Storage objects. Uses the appropriate parallelism (multi-process, multi-thread, both, or synchronous). Args: objects_to_delete: list of ObjectDeleteTask num_threads: int (optional), the number of threads to use. show_progress_bar: bool. If true, show a progress bar to the users when deleting files. z(Deleting {} {} from Google Cloud StorageobjectNrr)objects_to_deleterjru num_objectsrws r DeleteObjectsr}sF%&+ 6 = =T^^K: rs2'' 6#2-*)  3  ckk * !T!H t F!!Ht8%8$(;D.A"'4,2E$)6r