dZddlmZddlmZddlmZddlZddlZddlmZddlm Z ddlm Z Gd d e jZ y) zJImplements a file wrapper used for in-flight retries of streaming uploads.)absolute_import)division)unicode_literalsN)errors) hash_util) upload_streamcveZdZdZ d fd ZdZdZdZdZdZ d fd Z e jfd Z d ZxZS) BufferedUploadStreamzHSupports limited seeking within a non-seekable stream by buffering data.ct||d||tj|_||_d|_d|_d|_tj|j|_ y)aInitializes a FilePart instance. Args: stream (io.IOBase): The underlying stream wrapped by this class. max_buffer_size: Maximum size of the internal buffer. This should be >= to the chunk size used by the API to execute streaming uploads to ensure that at least one full chunk write can be repeated in the event of a server error. digesters (dict[util.HashAlgorithm, hashlib hash object]|None): See super class. progress_callback (func[int]|None): See super class. N)streamlength digestersprogress_callbackr) super__init__ collectionsdeque_buffer_max_buffer_size _buffer_start _position _buffer_endrcopy_digesters _digesters_checkpoint_digesters)selfr max_buffer_sizerr __class__s @lib/googlecloudsdk/command_lib/storage/buffered_upload_stream.pyrzBufferedUploadStream.__init__!sm" )T#+ $- $$&DL+DDDND!*!9!9$//!JDc|jSNrrs r_get_absolute_positionz+BufferedUploadStream._get_absolute_position@s >>r c||_yr"r#)roffsets r_update_absolute_positionz.BufferedUploadStream._update_absolute_positionCs DNr cg}|}|j|jkr|j}|jD]}|t |z|jk\rZ|j|z }t ||z }t ||}|j ||||z||z}|xj|z c_|t |z }dj|S)aGet any buffered data required to complete a read. If a backward seek has not happened, the buffer will never contain any information needed to complete a read call. Return the empty string in these cases. If the current position is before the end of the buffer, some of the requested bytes will be in the buffer. For example, if our position is 1, five bytes are being read, and the buffer contains b'0123', we will return b'123'. Two additional bytes will be read from the stream at a later stage. Args: amount (int): The total number of bytes to be read Returns: A byte string, the length of which is equal to `amount` if there are enough buffered bytes to complete the read, or less than `amount` if there are not. r )rrrrlenminappendjoin) ramount buffered_databytes_remainingposition_in_bufferdataoffset_from_positionbytes_to_read_this_block read_sizes r_read_from_bufferz&BufferedUploadStream._read_from_bufferFs(MO ~~(((--,,$ D )T^^ ;!%2D!D %(Y1E%E "2OD)   t$89M$-:. / 0 Y &/ ..I %.c$i' 88M ""r cy)a#Disables parent class digester checkpointing behavior. To guarantee that seeks within the buffer are possible, we need to ensure that the checkpoint is aligned with the buffer's start_byte. This is not possible if we save digester checkpoints when the parent class does so. Nr$s r_save_digesters_checkpointz/BufferedUploadStream._save_digesters_checkpointks r c|re|jj||xjt|z c_d}|j|jz |j kDr|jj }|xjt|z c_|r|j |j|jz z }|dk\r4|jj|| d|xj|zc_tj|j|dt||z |j|_ |j|jz |j kDryyy)axAdds data to the buffer, respecting max_buffer_size. The buffer can consist of many different blocks of data, e.g. [b'0', b'12', b'3'] With a maximum size of 4, if we read two bytes, we must discard the oldest data and keep half of the second-oldest block: [b'2', b'3', b'45'] Args: data (bytes): the data being added to the buffer. N) rr,rr*rrpopleft appendleftrupdate_digestersr_checkpoint_absolute_index)rr2 oldest_data refill_amounts r _store_dataz BufferedUploadStream._store_datats9  ll$ #d)#k   t11 1D4I4I Ill**,  c+.. //!3!335- a  LL # #K$@ A   - /   $ $((;3{+m;< >-1,>,>$ )   t11 1D4I4I I  r c.|duxs|dk}|r |j}n|}|j|}|t|z}|rt|d}n|rt||}nd}|xj t|z c_|j |||zS)zReads from the wrapped stream. Args: size: The amount of bytes to read. If omitted or negative, the entire stream will be read and returned. Returns: Bytes from the wrapped stream. Nrr )rr6r*r _get_datarrB)rsizeread_all_bytesr0r2new_datars rrEzBufferedUploadStream._get_datasT\-TAXN--oo  ! !/ 2Ds4y Oy$1"5h y$1/BhhNNc(m#NX (?r c|tjk(r[||jks||jkDr:t j dj ||j|j|}n|tjk(rt||jkDr/t j dj ||j|j|jr |j|jr|j|z}n%t j dj |||j|||_ y)z!Seeks within the buffered stream.zUnable to recover from an upload error because limited buffering is available for streaming uploads. Offset {} was requested, but only data from {} to {} is buffered.zNInvalid SEEK_END offset {} on streaming upload. Only {} bytes can be buffered.z;Invalid seek mode on streaming upload. Mode: {}, offset: {}N) osSEEK_SETrrrErrorformatSEEK_ENDabsrreadr_catch_up_digesters)rr'whence new_positions rseekzBufferedUploadStream.seeks!  $$$ $1A1A(All 44:F**D,<,<5>? ? l 2;;  Vt,, ,ll &vt/D/D EG G IId++ ,  IId++ ,^^f,l LL G N Nf   \*!DNr cy)ayIndicates that this stream is not seekable. Needed so that boto3 can correctly identify how to treat this stream. The library attempts to seek to the beginning after an upload completes, which is not always possible. Apitools does not check the return value of this method, so it will not raise issues for resumable uploads. Returns: False Fr8r$s rseekablezBufferedUploadStream.seekables r )NN)rD)__name__ __module__ __qualname____doc__rr%r(r6r9rBrErJrKrTrV __classcell__)rs@rr r sHP !% K>##J "?H@!# "8 r r ) rZ __future__rrrrrJ"googlecloudsdk.command_lib.storagerrr UploadStreamr r8r rr_s7 Q&' 58<C=55Cr