tdZddlmZddlmZddlmZddlmZddlZddlZddlm Z ddl m Z dd l m Z e jZGd d eZGd d ej"Zy)zIThreading code for estimating total work of long-running gsutil commands.)absolute_import)print_function)division)unicode_literalsN)thread_message) constants)parallelism_framework_utilceZdZdZddZy)SeekAheadResultzResult class for seek_ahead_iterator results. A class is used instead of a namedtuple, making it easier to document and use default keyword arguments. c ||_||_y)aCreate a SeekAheadResult. Args: est_num_ops: Number of operations the iterated result represents. Operation is loosely defined as a single API call for a single object. The total number of API calls may not be known at the time of iteration, so this number is approximate. data_bytes: Number of data bytes that will be transferred (uploaded, downloaded, or rewritten) for this iterated result. N) est_num_ops data_bytes)selfr rs *platform/gsutil/gslib/seek_ahead_thread.py__init__zSeekAheadResult.__init__'s#D DON)r)__name__ __module__ __qualname____doc__rrrr r s  !rr c(eZdZdZfdZdZxZS)SeekAheadThreadaThread to estimate total work to be performed by all processes and threads. Because the ProducerThread can only buffer a certain number of tasks on the global task queue, it cannot reliably provide the total count or size of iterated results for operations involving many iterated arguments until it nears the end of iteration. This thread consumes an iterator that should be functionally identical to the ProducerThread, but iterates to the end without adding tasks to the global task queue in an effort to estimate the amount of total work that the call to Apply will perform. It should be used only for large operations, and thus it is created by the main ProducerThread only when the number of iterated arguments exceeds a threshold. This thread may produce an inaccurate estimate if its iterator produces different results than the iterator used by the ProducerThread. This can happen due to eventual listing consistency or due to the source being modified as iteration occurs. This thread estimates operations for top-level objects only; sub-operations (such as a parallel composite upload) should be reported via the iterator as a single object including the total number of bytes affected. ctt| ||_||_||_d|_|jy)ayInitializes the seek ahead thread. Args: seek_ahead_iterator: Iterator matching the ProducerThread's args_iterator, but returning only object name and/or size in the result. cancel_event: threading.Event for signaling the seek-ahead iterator to terminate. status_queue: Status queue for posting summary of fully iterated results. FN)superrr status_queueseek_ahead_iterator cancel_event terminatestart)rrrr __class__s rrzSeekAheadThread.__init__Os: /4)+$D2D$DDNJJLrc d}d} |jD]`}|jry|tjzdk(r|jj ry||j z }||jz }b |jj ryt|jtj||tjy#t$r }Yd}~yd}~wwxYw)Nr)rr rNUM_OBJECTS_PER_LIST_PAGErisSetr rOSError_PutToQueueWithTimeoutrrSeekAheadMessagetime)r num_objectsnum_data_bytesseek_ahead_resultes rrunzSeekAheadThread.runbsKN #77  >>  )== =! C    $ $ & (444 +666 8(    '' ^(,  56    sC1C C C"C")rrrrrr. __classcell__)r"s@rrr6s0&6rr)r __future__rrrr threadingr)gslibr gslib.utilsrr PutToQueueWithTimeoutr'objectr Threadrrrrr7sRP&%' !23II!f!,J6i&&J6r