8 dZddlmZddlmZddlmZddlZddlZddlZddlm Z m Z m Z m Z m Z mZddlmZddlmZd Zd Zd d d dddddZej+Dcic]\}}|| c}}Zej/ZdedefdZdedefdZdedefdZdededefdZdedefdZdGdZddd edefd!Z d"e ede ee ee ee ffffd#Z!Gd$d%e Z"d&e ed"ede"fd'Z#d(Z$d)Z%d*ede efd+Z&d,Z'd-Z(d.Z)d/Z* dHd0e e+d1e ed2e efd3Z,Gd4d5eZ- dHd6ej\j^d ed7e e+d1e ed8e ede-f d9Z0 dId2e ede e efd:Z1 dHd6ed;e ed0e e+d1e ede efd?Z3d@edAe ee ffdBZ4dCedDe ee fdAe ee fdEefdFZ5ycc}}w)Jz#Bigquery Client library for Python.)absolute_import)division)print_functionN)AnyDictList NamedTupleOptional TypedDict)bq_error) bq_id_utilsizgs://cloudSqlawsazure sqlDataSource cloudSpanner cloudResourcespark) CLOUD_SQLAWSAzureSQL_DATA_SOURCE CLOUD_SPANNERCLOUD_RESOURCESPARK iam_role_idreturnc d|zS)zReturns properties for a connection with IAM role id. Args: iam_role_id: IAM role id. Returns: JSON string with properties to create a connection with IAM role id. z#{"accessRole": {"iamRoleId": "%s"}})rs 'platform/bq/utils/bq_processor_utils.pyMakeAccessRolePropertiesJsonr!)s / << tenant_idc d|zS)zReturns properties for a connection with tenant id. Args: tenant_id: tenant id. Returns: JSON string with properties to create a connection with customer's tenant id. z{"customerTenantId": "%s"}r)r#s r MakeTenantIdPropertiesJsonr%6s & 11r"federated_app_client_idc d|zS)a Returns properties for a connection with a federated app (client) id. Args: federated_app_client_id: federated application (client) id. Returns: JSON string with properties to create a connection with customer's federated application (client) id. z&{"federatedApplicationClientId": "%s"}r)r&s r +MakeAzureFederatedAppClientIdPropertiesJsonr(Ds 24K KKr"cd|d|dS)a1Returns properties for a connection with tenant and federated app ids. Args: tenant_id: tenant id federated_app_client_id: federated application (client) id. Returns: JSON string with properties to create a connection with customer's tenant and federated application (client) ids. z{"customerTenantId": "z%", "federatedApplicationClientId" : "z"}r)r#r&s r 4MakeAzureFederatedAppClientAndTenantIdPropertiesJsonr*Ts r"namec2tjdd|S)z-Convert a name with underscores to camelcase.z_[a-z]cF|jddjS)Nr)groupupper)matchs r zToLowerCamel..ls Aq(9(?(?(Ar")resub)r+s r ToLowerCamelr5js A4 HHr"c P|jd|jDy)zAdds all kwds to config dict, adjusting keys to camelcase. Note this does not remove entries that are set to None, however. Args: config: A configuration dict. **kwds: A dict of keys and values to set in the config. c3BK|]\}}| t||fywN)r5).0kvs r z"ApplyParameters..xs"OAa!$s N)updateitems)configkwdss r ApplyParametersrAos --OOOr"project_referencez,bq_id_utils.ApiClientHelper.ProjectReferencelocationc,d|jzdz|zS)aFormats a project identifier for data transfers. Data transfer API calls take in the format projects/(projectName), so because by default project IDs take the format (projectName), add the beginning format to perform data transfer commands Args: project_reference: The project id to format for data transfer commands. location: The location id, e.g. 'us' or 'eu'. Returns: The formatted project name for transfers. z projects/z /locations/) projectId)rBrCs r #FormatProjectIdentifierForTransfersrF{s $ (22 2] BX MMr" json_stringc tj|S#t$r }tjd|d|d}~wwxYw)zAWrapper for standard json parsing, may throw BigQueryClientError.z Error decoding JSON from string : N)jsonloads ValueErrorr BigqueryClientError)rGes r ParseJsonrOsA ::k ""   & &4?C s A;Ac(eZdZUeeed<eed<y) InsertEntry insert_idrecordN)__name__ __module__ __qualname__r str__annotations__objectrr"r rQrQs c] .r"rQrRc tj|}t|tst j dt ||S#t$r&}t j dt|d}~wwxYw)zParses a JSON encoded record and returns an InsertEntry. Arguments: insert_id: Id for the insert, can be None. json_string: The JSON encoded data to be converted. Returns: InsertEntry object for adding to a table. zValue is not a JSON objectzCould not parse object: N) rJrK isinstancedictr rMrQrLrW)rRrGrowrNs r JsonToInsertEntryr^siQ **[ !C c4  ( ()E FF y# && Q  & &s1v'O PPQsAA A7!A22A7cN|jdi}d|vrd|dvr|ddSy)zHelper to return the session id if the job is part of one. Args: job: a job resource to get statistics and sessionInfo from. Returns: sessionId, if the job is part of a session. statistics sessionInfo sessionIdN)get)jobstatss r GetSessionIdrfs; '', #%e u]/C C   ,, r"ctd} t|jdijj|j S#t $rYywxYw)zHelper for job printing code.)extractloadquerycopy configurationN)setrckeys intersectionpopKeyError)job_info job_namess r GetJobTypeNamertsX67) HLL" - 2 2 45 i   sAA AA source_stringc|jdDcgc]}|j}}|Dcgc]}|jts|}}|st j d|rAt |t |k7r(t j djt|S|d}t |dkDr"t j dt |fztjj|st j d|tjj|st j d||Scc}wcc}w) aTake a source string and return a list of URIs. The list will consist of either a single local filename, which we check exists and is a file, or a list of gs:// uris. Args: source_string: A comma-separated list of URIs. Returns: List of one or more valid URIs, as strings. Raises: bq_error.BigqueryClientError: if no valid list of sources can be determined. ,zNo sources specifiedz(All URIs must begin with "{}" if any do.rr.z7Local upload currently supports only one file, found %dzSource file not found: zSource path is not a file: ) splitstrip startswithGCS_SCHEME_PREFIXr rMlenformatospathexistsisfile)rusourcesourcesgs_uriss r ProcessSourcesrsI +8*=*=c*B C*BV\\^*B' C" "f&7&78I&Jf7     & &'= >>  7|s7|#  ( ( 4 ; ;>& !  ( ()/ 1  77>>& !  ( (-3 5  .7 D sEE Ec*|jddS)z#Convert a kind to just a type name.#) partition)kinds r KindToNamers  Q r"cLtjD] \}}||vs |cSyr8)CONNECTION_TYPE_TO_PROPERTY_MAPr>) connectiontps r GetConnectionTypers*-335daJ h6 r"cd|vr+t|d}|dz}||vrdtd|d|d|d}|Dcgc]}|jds|}}t|dk7rtdt|d||d }|d j |dd z}t t j|d }|td ||jd i||Scc}w) z-Construct a Reference from a server response.r Referencez Cannot find z in object of type rIr.zExpected one Reference, found rNzUnknown reference type: r) rrLendswithr|r0getattrr ApiClientHelperCreate) object_infotypename lower_camelr:rn upper_camelreference_types r ConstructObjectReferencer s {+f-.H[(K+%  (K 1  H" >{!ajj&=A{D > 4yA~ 47It D q'KA$$&QR8+;66 TJ. X? @@    :[!9 :: ?s C Ccz|jj}|dj|ddz}|t|iS)z,Construct an Object from an ObjectReference.rr.N) __class__rTlowerr\) referencerrs r ConstructObjectInfor&s?  ) )( !!#hqrl2+ tI ''r" max_results page_tokenfilter_expressioncFt|}|||d<|||d<|||d<|S)#Create and populate a list request. maxResultsfilter pageToken)r\)rrrrrequests r PrepareListRequestr-sB O''GL")GH%GK .r"cNeZdZUeed<eeed<eeed<eeeed<y)TransferListRequestparentpageSizer dataSourceIdsN)rTrUrVrWrXr intrrr"r rrCs* + SM c]$s)$$r"rr page_sizedata_source_idsctt||}|||d<|||d<|O|jd}|ddk(r|djd}||d<|Stjd |dz|S) rrrr:rrr.rwzSInvalid filter flag values: '%s'. Expected format: '--filter=dataSourceIds:id1,id2')r\rFrxr BigqueryError)rrCrrrrs r PrepareTransferListRequestrMs  0H E '#GJ%GK %++C0Oq_,'*005o!0go .  " " > A    .r"c|y|jdr$ |jddjdStjd|zdz#t$r!}tjd|zdz|d}~wwxYw) aEParses the state filter for list jobs. Args: filter_expression: A string containing the state filter, e.g., 'state:done'. Returns: A single state filter or a list of filters to apply. Returns None if no filter is provided. Raises: bq_error.BigqueryClientError: if the filter expression is invalid. Nzstates:rr.rwInvalid flag argument ""z*, the expression must start with "states:")rzrx IndexErrorr r)rrNs r ParseStateFilterExpressionrks !!),  $ $S )! , 2 23 77  !   6 7    " " #&7 7# =  s"A A>A99A> run_attemptstatesct|}||d<||tkDrt}||d<|t||d<|||d<|S)z0Create and populate a transfer run list request.r runAttemptrrr)r\ MAX_RESULTSr)rrrrrrs r PrepareTransferRunListRequestrs_  "'%',[ k%GJ 26:GH%GK .r" message_typecJt|}||tkDrt}||d<|||d<|Jd|vr+ |jddjd}||d <|St j d |zd z|S#t$r!}t j d |zd z|d}~wwxYw) z0Create and populate a transfer log list request.rNrrz messageTypes:rr.rw messageTypesrr)r\rrxrr r)rrrrrrNs r PrepareListTransferLogRequestrs  "'[ k%GJ%GK,&#))#.q177< ". .  " " #l 2S 8  .$$ % 4s :  s(A88 B"BB"paramsr>c tj|}||d<|S#t$r}tjd|d}~wwxYw)aProcesses the params flag. Args: params: The user specified parameters. The parameters should be in JSON format given as a string. Ex: --params="{'param':'value'}". items: The body that contains information of all the flags set. Returns: items: The body after it has been updated with the params flag. Raises: bq_error.BigqueryError: If there is an error with the given params. zWParameters should be specified in JSON format when creating the transfer configuration.Nr)rJrK Exceptionr r)rr> parsed_paramsrNs r ProcessParamsFlagrsT JJv&M "%/ ,    #    s A=Arefresh_window_daysdata_source_info data_sourcecd|vr'|ddk(r||d<|Stjd|ztjd|z)aProcesses the Refresh Window Days flag. Args: refresh_window_days: The user specified refresh window days. data_source_info: The data source of the transfer config. items: The body that contains information of all the flags set. data_source: The data source of the transfer config. Returns: items: The body after it has been updated with the refresh window days flag. Raises: bq_error.BigqueryError: If the data source does not support (custom) window days. dataRefreshTypeCUSTOM_SLIDING_WINDOWdata_refresh_window_daysz=Data source '%s' does not support custom refresh window days.z6Data source '%s' does not support refresh window days.)r r)rrr>rs r ProcessRefreshWindowDaysFlagrsg***)*.EE*=e &' l  " " I     @;N r")rN)NNNr8)6__doc__ __future__rrrrJr~r3typingrrrr r r utilsr r rr{rr>CONNECTION_PROPERTY_TO_TYPE_MAPrnCONNECTION_TYPESrWr!r%r(r*r5rArFrOrQr^rfrtrrrrrrrrrProjectReferencerrrrrr)rrs00r rs*&% CC    &#% #5::<#(,} d3iF"& $ ##  SM ."& $"& # 3- <c$sCx.2!!38n! S>! !i #s G