*dZddlmZddlmZddlmZddlZddlZddlZddlm Z ddlm Z ddl m Z ddlmZdd lmZdd lmZddlm Z dd lmZdd lmZdd lmZddlZdZdZej:j<j>j@jCeej:j<j>jDjCeddZ#ddZ$dZ%ddZ&ddZ'ddZ(dZ)Gdde*Z+Gdde*Z,y)zHelpers for accessing GCS. Bulk object uploads and downloads use methods that shell out to gsutil. Lightweight metadata / streaming operations use the StorageClient class. )absolute_import)division)unicode_literalsN) exceptions)transfer) storage_api) storage_util)apis)log) properties) resourcesgs<ctjjjj }|r t ||yt |||y)N)storage_client)r VALUESstorage use_gsutilGetBool _UploadGsutil_UploadStorageClient)files destinationrrs 6lib/googlecloudsdk/api_lib/dataproc/storage_helpers.pyUploadr8s=  ((33;;=*%% NKc |xstj}|D]s}tjj |}tjj ||}t jj|} |j||uy#tj$r:}tjdjdj |||d}~wwxYw)zUpload a list of local files to GCS. Args: files: The list of local files to upload. destination: A GCS "directory" to copy the files into. storage_client: Storage api client used to copy files to gcs. z)Failed to upload files ['{}'] to '{}': {}', 'N)r StorageClientospathbasenamejoinr ObjectReferenceFromUrl CopyFileToGCSrBadFileException dp_exceptionsFileUploadErrorformat) rrrclientfile_to_upload file_namedest_url dest_objecterrs rrrBs  8[668&n  0Iww||K3H..66x@K5 >;7   & &5  ) ) 5 < <kk% +s 4 555s=BC%5CCc|}||gz }tjd|}|dk7r4tjdj dj ||y)zUpload a list of local files to GCS. Args: files: The list of local files to upload. destination: A GCS "directory" to copy the files into. cprz5Failed to upload files ['{0}'] to '{1}' using gsutil.rN)r RunGsutilCommandr(r)r*r#)rrargs exit_codes rrrWsb $;-$++D$7)!^  ' '?FF KK   - ..rc|xstj} |j|S#tj$rYywxYw)zGets a bucket if it exists. Args: bucket: The bucket name. storage_client: Storage client instance. Returns: A bucket message, or None if it doesn't exist. N)rr GetBucketBucketNotFoundError)bucketrr+s rr7r7gsE  8[668&   F ##  ( ( s+AAc\|xstj}|j|||y)amCreates a bucket. Creates a bucket in the specified region. If the region is None, the bucket will be created in global region. Args: bucket: Name of bucket to create. region: Region to create bucket in. storage_client: Storage client instance. project: The project to create the bucket in. If None, current Cloud SDK project is used. )locationprojectN)rrCreateBucketIfNotExists)r9regionrr<r+s rr=r=ys,  8[668&  &' JrcX|xstj}tjj |} |j |}t j|d}|jS#tj$r%tjdj|wxYw)aKReads an object's content from GCS. Args: object_url: The URL of the object to be read. Must have "gs://" prefix. storage_client: Storage api client used to read files from gcs. Raises: ObjectReadError: If the read of GCS object is not successful. Returns: A str for the content of the GCS object. zutf-8)encodingzFailed to read file '{0}'.)rrr r$r% ReadObjectio TextIOWrapperreadrr'r(ObjectReadErrorr*) object_urlrr+ object_refbytes_iowrappers rrArAs  8[668&++33J?*9  ,Hx':G <<>  $ $9  ' '$++J7 999s 7A118B)ctjj|}|j|j|j S)z.Build an Object proto message from a GCS path.)r9name)r REGISTRYParseStorageURLObjectr9object)r!messagesresources r GetObjectRefrRs3    / / 5( hoo FFrc*eZdZdZdZddZdZdZy)rzMicro-client for accessing GCS.cptjdd|_tjdd|_y)Nrv1) core_apisGetClientInstancer+GetMessagesModulerPselfs r__init__zStorageClient.__init__s*--i>DK// 4@DMrNc|jj|j|j} |jj j ||S#tj$rYywxYw)N)r9rO)requestdownload) rPStorageObjectsGetRequestr9rKr+objectsGetapitools_exceptionsHttpNotFoundError)rZrGr^r]s r _GetObjectzStorageClient._GetObjectsgmm44  5:G [[ $ $Wx $ HH  0 0 s&AA0/A0c$|j|S)a-Get the object metadata of a GCS object. Args: object_ref: A proto message of the object to fetch. Only the bucket and name need be set. Raises: HttpError: If the responses status is not 2xx or 404. Returns: The object if it exists otherwise None. )rd)rZrGs r GetObjectzStorageClient.GetObjects ??: &&rctjj||jd}|j |||S)aBuild an apitools Download from a stream and a GCS object reference. Note: This will always succeed, but HttpErrors with downloading will be raised when the download's methods are called. Args: stream: An Stream-like object that implements write() to write into. object_ref: A proto message of the object to fetch. Only the bucket and name need be set. Returns: The download. F) total_size auto_transfer)r^)rDownload FromStreamsizerd)rZstreamrGr^s rBuildObjectStreamzStorageClient.BuildObjectStreams@  ++:??%,AHOOJO2 OrN)__name__ __module__ __qualname____doc__r[rdrfrnrrrrs'A' rrcXeZdZdZd dZedZdZdZdZ e jfdZ y) StorageObjectSeriesStreamzFI/O Stream-like class for communicating via a sequence of GCS objects.Ncb||_|xs t|_d|_d|_d|_y)a+Construct a StorageObjectSeriesStream for a specific gcs path. Args: path: A GCS object prefix which will be the base of the objects used to communicate across the channel. storage_client: a StorageClient for accessing GCS. Returns: The constructed stream. TrN) _base_pathr_gcs_open_current_object_index_current_object_pos)rZr!rs rr[z"StorageObjectSeriesStream.__init__s4DO1-/DIDJ"#D !Drc|jS)zWhether the stream is open.rzrYs ropenzStorageObjectSeriesStream.opens ::rcd|_y)zClose the stream.FNr~rYs rClosezStorageObjectSeriesStream.Closes DJrc2|js tdy)NzI/O operation on closed stream.)r ValueErrorrYs r _AssertOpenz%StorageObjectSeriesStream._AssertOpens 99 8 99 rcdj|j|}|jjt ||jj S)z!Get the ith object in the series.z {0}.{1:09d})r*rxryrfrRrP)rZir!s rrdz$StorageObjectSeriesStream._GetObjects>    3D 99  |D$))2D2DE FFrcn|jd}d}|}||kra|j|jdz}|r|r! |j|j}|s |S|j|jz }|dkr$tdj|j|jdk(r|j |S||z } t| |} | dkDrb|jj||} | j!|j|j| zdz |xj| z c_|| z }|xr|j|jk(} | r |}|xjdz c_d|_d |S|S#tj$r!}t j d|Yd}~|Sd}~wwxYw)aRead from this stream into a writable. Reads at most n bytes, or until it sees there is not a next object in the series. This will block for the duration of each object's download, and possibly indefinitely if new objects are being added to the channel frequently enough. Args: writable: The stream-like object that implements write() to write into. n: A maximum number of bytes to read. Defaults to sys.maxsize (usually ~4 GB). Raises: ValueError: If the stream is closed or objects in the series are detected to shrink. Returns: The number of bytes read. rNzFailed to fetch GCS output: %szObject [{0}] shrunk.)rrdr{rb HttpErrorr warningrlr|rr*rKrminryrnGetRange) rZwritablen bytes_read object_infomax_bytes_to_readnext_object_infoerrornew_bytes_availablebytes_left_to_readnew_bytes_to_readr^object_finisheds rReadIntoWritablez*StorageObjectSeriesStream.ReadIntoWritable s* JK ( ()C)Ca)GH, (B(BC+ L I(,,t/G/GG q /66{7G7GHII   Q   8 5-z902EF Q 99..xE  $ $  $ $'8 81 < >   $55 ''  Kt77;;K;KK &  ""a'"#$   :W#,,  ++7 ? R W sFF4F//F4ro) rprqrrrsr[propertyrrrrdsysmaxsizerrtrrrvrvs=N!*  :G *-Lrrvro)NN)-rs __future__rrrrBr rapitools.base.pyrrbrgooglecloudsdk.api_lib.dataprocr(googlecloudsdk.api_lib.storagerr googlecloudsdk.api_lib.utilr rVgooglecloudsdk.calliopegooglecloudsdk.corer r r six.moves.urllib.parsesixSTORAGE_SCHEME HTTP_TIMEOUTmovesurllibparse uses_relativeappend uses_netlocrrrr7r=rArRrOrrvrtrrrs  '' >%G679.#*)  $$++N; "")).9L5* . $K"92G 4F4nvvr