4 dZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl m Z m Z m Z mZmZmZddlZddlmZddlmZddlZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z ddlm!Z! dsdejDde e#e#fdee$dee$fdZ%dejDfdZ& dtdejDdee jNjPdee$dee#deeee#e#fdee#dee#dee)dee#fdZ* dtdejDdee jNjPdee$dee#deeee#e#fdee#dee#dee)dee#fdZ+ dudejDd ee jNjXd!e jNjXd"ee#d#ee#d$ee)d%ee#d&ee#fd'Z-dejDde jNj\d(e)fd)Z/ dvdejDde jNj\d*ee)fd+Z0 dwdejDd,ee#d-ee#d.ee#d/ee#f d0Z1 dxdejDd1e#d2ee)d3ee)d4ee)d5ee#d6ee$dee$d7ee$d8ee$d9ee$d:ee2d,ee#d;ee)d/ee#dee)f$d?Z4 dydejDd.ee#d,ee#dee$d7ee$d/ee#f d@Z5 dwdejDd,ee#d-ee#d.ee#d/ee#f dAZ6 dydejDdBee)d,ee#d-ee#d.ee#d/ee#f dCZ7 dzdejDd,ee#d.ee#d/ee#fdDZ8dEejrdfdejDdFe jNj\dGe#dHe$dIee gejtff dJZ; d{dejDdFe jNj\dGe#dHe$fdKZ d|dejDd1e#dOee#d"ee#d#ee#dPee#d4ee)dQee)d2ee)d3ee)d:ee2dRee)dSee$d6ee$d;ee)dTeee#dUee e#e#fd8ee$d9ee$d>ee)dt|tr|j}n|D cgc]} | j}} t j| d|||||| d<||| d<|jjjdi| j!} | j#dg} |d | vrt%| |kr|t%| z | d <| d | d <|jjjdi| j!} | j'| j#dgd | vrt%| |krt)| } d | vr| d | d <d| vr| d| d<| Scc} w)aReturn a list of jobs. Args: bqclient: A BigqueryClient to get state and request clients from. reference: The ProjectReference to list jobs for. max_results: The maximum number of jobs to return. page_token: Current page token (optional). state_filter: A single state filter or a list of filters to apply. If not specified, no filtering is applied. min_creation_time: Timestamp in milliseconds. Only return jobs created after or at this timestamp. max_creation_time: Timestamp in milliseconds. Only return jobs created before or at this timestamp. all_users: Whether to list jobs for all users of the project. Requesting user must be an owner of the project to list all jobs. parent_job_id: Retrieve only child jobs belonging to this parent; None to retrieve top-level jobs. Returns: A dict that contains enytries: 'results': a list of jobs 'token': nextPageToken for the last page, if present. ) id_fallbacksr5r2methodfull) projectionr8r;r<minCreationTimemaxCreationTimejobs nextPageToken maxResults pageToken)r>token unreachabler)bq_client_utilsNormalizeProjectReferencer typecheckrProjectReferencer MAX_RESULTSPrepareListRequest isinstancestrlowerApplyParametersr#rHr/executegetlenextenddict)rr5r6r7r8r9r:r;r<requestsresultr>responses r*r?r?s F77y)!!22  '333&22k  1 1j ',$!'')l)56Aaggil6$$ ! "!2G "!2G  )8   " " $ ) ) 4G 4 < < >& JJvr "' V #G {(B)CL8gl#O4gk-x!!&&(--88@@Bf nnVZZ+, V #G {(B ' "(/HWf$]3H] /97s+G5source_referencesdest_referencecreate_dispositionwrite_dispositionignore_already_existsoperation_typedestination_expiration_timec  |D]2} tj| tjjd4tj|tjjdt ||D cgc] } t | c} d} |r|| d<|r|| d<|r|| d<t j | || t|d| ifi| Scc} w#tj$r} |rYd } ~ y | d } ~ wwxYw) aCopies a table. Args: bqclient: A BigqueryClient to get state and request clients from. source_references: TableReferences of source tables. dest_reference: TableReference of destination table. create_disposition: Optional. Specifies the create_disposition for the dest_reference. write_disposition: Optional. Specifies the write_disposition for the dest_reference. ignore_already_exists: Whether to ignore "already exists" errors. encryption_configuration: Optional. Allows user to encrypt the table from the copy table command with Cloud KMS key. Passed as a dictionary in the following format: {'kmsKeyName': 'destination_kms_key'} **kwds: Passed on to ExecuteJob. Returns: The job description, or None for ignored errors. Raises: BigqueryDuplicateError: when write_disposition 'WRITE_EMPTY' is specified and the dest_reference table already exists. CopyTablerB)destinationTable sourceTables"destinationEncryptionConfiguration operationTypedestinationExpirationTime)rcrdcopyN) rrPrTableReferencer\rrW ExecuteJobrBigqueryDuplicateError) rrarbrcrdreencryption_configurationrfrgr3src_ref copy_configes r*ririsF#g##22# !!00  ~.4EF4EtG}4EF+8PK45#1K  /JK+,$$+)  h 5 > >>%G&  ( (   G s$6C ;CC0"C+)C++C0returnctj|tjjd |jj j dit|jS#tj$rYywxYw)zReturns true if the job exists. JobExistsrBFr) rrPrrr#rHrYr\rXrBigqueryNotFoundError)rr5s r*ryry"st ,,99+ (8   " " $ ( ( ;4 ? ; C C EE  ' ' sAA33B B ignore_not_foundc tj|tjjd |jj j dit|jy#tj$r|sYywxYw)aDeletes JobReference reference. Args: bqclient: A BigqueryClient to get state and request clients from. reference: the JobReference to delete. ignore_not_found: Whether to ignore "not found" errors. Raises: BigqueryTypeError: if reference is not a JobReference. bq_error.BigqueryNotFoundError: if reference does not exist and ignore_not_found is False. DeleteJobrBNr) rrPrrr#rHdeleter\rXrrz)rr5r{s r*r}r}0sy",,99+ $H$$7tI7??A  ' '    sAA44B  B  project_id upload_filejob_idlocationc|xs |j}|stjd|j}|jrt d|jD|d<d|i}|xs |j }t|tjr|j|}|||d}||d<|r||d<d}|rH|j} tj|jd k(rd } tj |d | }|j"j%j'||| } |r rt)j*| } | S| j-} | S)aStart a job with the given configuration. Args: bqclient: A BigqueryClient to get state and request clients from. configuration: The configuration for a job. project_id: The project_id to run the job under. If None, bqclient.project_id is used. upload_file: A file to include as a media upload to this request. Only valid on job requests that expect a media upload file. job_id: A unique job_id to use for this job. If a JobIdGenerator, a job id will be generated from the job configuration. If None, a unique job_id will be created for this request. location: Optional. The geographic location where the job should run. Returns: The job resource returned from the insert job request. If there is an error, the jobReference field will still be filled out with the job reference used in the request. Raises: bq_error.BigqueryClientConfigurationError: if project_id and bqclient.project_id are None. z(Cannot start a job without a project id.c3JK|]}|jddddyw)=rN) partition).0props r* zStartJob..ws)'.CdsADqD!.Cs!# properties configuration)jobId projectId jobReferencerrFzapplication/octet-stream)filenamemimetype resumable)body media_bodyr)rr BigqueryClientConfigurationErrorro job_propertyr\job_id_generatorrTrNJobIdGeneratorGenerateenable_resumable_uploadsosstatst_size http_requestMediaFileUploadr#rHinsertrexecute_in_chunks_with_progressrX) rrrrrr job_request job_reference media_uploadrr]r_s r*StartJobrQss>0X00*   3 32  $$&- "&'.6.C.C'#M,!-0+  .X..&667 __] +F $:>M"/K"*mJ,11I ww{##q(i//+L    # # % , , <: - 'Y  9 9' BF -__ F -r,querydry_run use_cachepreserve_nulls request_idmaximum_bytes_billed timeout_msjob_timeout_ms max_slotsmin_completion_ratiouse_legacy_sqljob_creation_modereservation_idcreate_sessionc | xs |j} | stjdd|i}| r| |d<|r||d<|jr"t j |j|d<|7t jjrttj}t||}tj|fid|d|d |d |d |d | d | d|d|d| d|d|d|d|d|d|d|dn|rdndtj||tj||tj d|| ||j"j%j&d|| d|j)S)a Executes the given query using the rpc-style query api. Args: bqclient: A BigqueryClient to get state and request clients from. query: Query to execute. dry_run: Optional. Indicates whether the query will only be validated and return processing statistics instead of actually running. use_cache: Optional. Whether to use the query cache. Caching is best-effort only and you should not make assumptions about whether or how long a query result will be cached. preserve_nulls: Optional. Indicates whether to preserve nulls in input data. Temporary flag; will be removed in a future version. request_id: Optional. The idempotency token for jobs.query maximum_bytes_billed: Optional. Upper limit on the number of billed bytes. max_results: Maximum number of results to return. timeout_ms: Timeout, in milliseconds, for the call to query(). job_timeout_ms: Optional. How long to let the job run. max_slots: Optional. Cap on target rate of slot consumption by the query. min_completion_ratio: Optional. Specifies the minimum fraction of data that must be scanned before a query returns. This value should be between 0.0 and 1.0 inclusive. project_id: Project id to use. external_table_definitions_json: Json representation of external table definitions. udf_resources: Array of inline and external UDF code resources. use_legacy_sql: The choice of using Legacy SQL for the query is optional. If not specified, the server will automatically determine the dialect based on query information, such as dialect prefixes. If no prefixes are found, it will default to Legacy SQL. location: Optional. The geographic location where the job should run. connection_properties: Optional. Connection properties to use when running the query, presented as a list of key/value pairs. A key of "time_zone" indicates that the query will be run with the default timezone corresponding to the value. job_creation_mode: Optional. An option for job creation. The valid values are JOB_CREATION_REQUIRED and JOB_CREATION_OPTIONAL. reservation_id: Optional. An option to set the reservation to use when execute the job. Reservation should be in the format of "project_id:reservation_id", "project_id:location.reservation_id", or "reservation_id". create_session: Optional. True to create a session for the query. query_parameters: parameter values for use_legacy_sql=False queries. positional_parameter_mode: If true, set the parameter mode to POSITIONAL instead of the default NAMED. destination_encryption_configuration: Optional. Allows user to encrypt the table created from a query job with a Cloud KMS key. **kwds: Extra keyword arguments passed directly to jobs.Query(). Returns: The query response. Raises: bq_error.BigqueryClientConfigurationError: if project_id and bqclient.project_id are None. bq_error.BigqueryError: if query execution fails. z(Cannot run a query without a project id.rtableDefinitionsuserDefinedFunctionResourcesdefaultDatasetNrrruse_query_cacherrrr6rrr reservationrrquery_parameters$destination_encryption_configurationparameter_mode POSITIONALNAMEDconnection_properties)rz3Calling bqclient.apiclient.jobs().query(%s, %s, %s))rrr)rrr dataset_idrNGetQueryDefaultDatasetr rjobs_query_use_request_idrUuuiduuid4_GetReservationPathrrWloggingdebugr#rHrrX)rrrrrrrr6rrrrrexternal_table_definitions_json udf_resourcesrrrrrrrpositional_parameter_moderr3r]reservation_paths r*_StartQueryRpcrsj0X00*   3 32 e '$"AG .;G *+  / F F!G  EKKAATZZ\"J( $$ #0     $$0*#$ (!",P#& # *5L7),$$ %:$$Wg> --;   h u9Z9379wyr,c|xs |j}|stjdi}tj|||||||j j jdi|jS)aPWaits for a query job to run and returns results if complete. By default, waits 10s for the provided job to complete and either returns the results or a response where jobComplete is set to false. The timeout can be increased but the call is not guaranteed to wait for the specified timeout. Args: bqclient: A BigqueryClient to get state and request clients from. job_id: The job id of the query job that we are waiting to complete. project_id: The project id of the query job. max_results: The maximum number of results. timeout_ms: The number of milliseconds to wait for the query to complete. location: Optional. The geographic location of the job. Returns: The getQueryResults() result. Raises: bq_error.BigqueryClientConfigurationError: if project_id and bqclient.project_id are None. z.Cannot get query results without a project id.)rrrr6rr) rrrrrWr#rHgetQueryResultsrX)rrrr6rrr3s r*GetQueryResultsr2s<0X00*   3 38  $$$   3   " 2 2 :T : B B DDr,ct||||||}|dddk7r!tj|}t||}t j |S)aStarts a job and waits for it to complete. Args: bqclient: A BigqueryClient to get state and request clients from. configuration: The configuration for a job. project_id: The project_id to run the job under. If None, bqclient.project_id is used. upload_file: A file to include as a media upload to this request. Only valid on job requests that expect a media upload file. job_id: A unique job_id to use for this job. If a JobIdGenerator, a job id will be generated from the job configuration. If None, a unique job_id will be created for this request. location: Optional. The geographic location where the job should run. Returns: job, if it did not fail. Raises: BigQueryError: if the job fails. rrrrstatusstateDONE)rrr1WaitJobrNRaiseIfJobError)rrrrrrr_rs r*RunJobSynchronouslyrasa8    & Hg&(&??GM X} -F  ( ( 00r,syncc| |j}|rt||||||}|St||||||}tj||S)z,Execute a job, possibly waiting for results.r)rrrrNr)rrrrrrrjobs r*rqrqsk \ ==D   C$ *   C##C( *r,c|xs |j}|stjd|stjdtjj j |||}|jjjd it|jd}|dddk7r.|jr"tj|}t||}|S) a Attempt to cancel the specified job if it is running. Args: bqclient: A BigqueryClient to get state and request clients from. project_id: The project_id to the job is running under. If None, bqclient.project_id is used. job_id: The job id for this job. location: Optional. The geographic location of the job. Returns: The job resource returned for the job for which cancel is being requested. Raises: bq_error.BigqueryClientConfigurationError: if project_id or job_id are None. z)Cannot cancel a job without a project id.z%Cannot cancel a job without a job id.)rrrrrrr)rr)rrrrrrrr#rHcancelr\rXrrr1r)rrrrrr_s r* CancelJobrs,0X00*   3 33    3 3/ --::AA&8B-'h&&=m)<=EEGN  Hg&(X]]&??GM X] ;F -r,rrrwaitwait_printer_factoryctj|tjjdt j}d}|r|}n|j }t jt jddtdddt jd}d } d } d } | |kr t|||| \} }|d d} d } | r|j|j| | nw tt'|D]K}t j|z } |j|j| | t j(dM| |krt+d| |j-|S#tj$r } tj d| Yd} ~ d} ~ wtj"$r } tj d| Yd} ~ d} ~ wtj$$r} | rd} Yd} ~ d} ~ wwxYw)aPoll for a job to run until it reaches the requested status. Arguments: bqclient: A BigqueryClient to get state and request clients from. job_reference: JobReference to poll. status: (optional, default 'DONE') Desired job status. wait: (optional, default maxint) Max wait time. wait_printer_factory: (optional, defaults to bqclient.wait_printer_factory) Returns a subclass of WaitPrinter that will be called after each job poll. Returns: The job object returned by the final status call. Raises: StopIteration: If polling does not reach the desired state before timing out. ValueError: If given an invalid wait value. rrBNrrUNKNOWNF)rrrrz+Transient error during job status check: %sTz1Wait timed out. Operation not finished, in state )rrPrrtimer itertoolschainrepeatrangePollJobprintrrBigqueryCommunicationErrorrwarningBigqueryBackendErrorBigqueryServiceErrornextsleep StopIterationdone)rrrrr start_timerprinterwaits current_waitcurrent_statusin_error_staterrv_s r*rrs6!!..  yy{* #"$G++-G //aU1b!_b %,..(M&tLidC8}W-nn  m))<H  "4; YY[:-l mmM''~F jjm - 6     ,,. *1  . .H ooCQGG  ( (H ooCQGG  ( ( n s06;E;;HF))H?GH0G;;Hc,tj|tjjdt j |}|j jjdit|j}|dd}||k(|fS)aPoll a job once for a specific status. Arguments: bqclient: A BigqueryClient to get state and request clients from. job_reference: JobReference to poll. status: (optional, default 'DONE') Desired job status. wait: (optional, default 0) Max server-side wait time for one poll call. Returns: Tuple (in_state, job) where in_state is True if job is in the desired state. Raises: ValueError: If given an invalid wait value. rrBrrr) rrPrrrN NormalizeWaitr#rHrYr\rX)rrrrrcurrents r*rr-s*!!..   & &t ,$%!%%<](;<DDF# M' "' V S !!r,c ^t|}d|d<t|fi|}t||d||S)aZRun a query job synchronously, and return the result. Args: bqclient: A BigqueryClient to get state and request clients from. start_row: first row to read. max_rows: number of rows to read. **kwds: Passed on to Query. Returns: A tuple where the first item is the list of fields and the second item a list of rows. Trr)rr)r\Queryr+)rrrr3new_kwdsrs r*RunQueryrRsD$$Z((6 h#(##  .  r,max_single_waitc |jstjd|stjd|/tjj stjd| r| }n|j }tj}d}d}d} |dntj|z }||z }| -tt|| dz}|dkrtj}|dkr td||}|j.| |j}nt|jt|}td)id|d |d |d |d |d |d|d| d|d|d|d| d| d|d|d|d||}|rLttt|d}d|vr|d|dd d<d|vr|d|dd d<gg|fSd |vrdt!j"j$j&d)i|d }n7|j)|j*|d!t-||j*|||"}|d#r9t/||r t|nid||$\}}i} d%|vr|d%| d%<d&| |d'}|||fS #tj0$r }!t3j4d(|!Yd}!~!0d}!~!wtj6$r }!t3j4d(|!Yd}!~!ad}!~!wwxYw)*a Executes the given query using the rpc-style query api. Args: bqclient: A BigqueryClient to get state and request clients from. query: Query to execute. dry_run: Optional. Indicates whether the query will only be validated and return processing statistics instead of actually running. use_cache: Optional. Whether to use the query cache. Caching is best-effort only and you should not make assumptions about whether or how long a query result will be cached. preserve_nulls: Optional. Indicates whether to preserve nulls in input data. Temporary flag; will be removed in a future version. request_id: Optional. Specifies the idempotency token for the request. maximum_bytes_billed: Optional. Upper limit on maximum bytes billed. max_results: Optional. Maximum number of results to return. wait: (optional, default maxint) Max wait time in seconds. min_completion_ratio: Optional. Specifies the minimum fraction of data that must be scanned before a query returns. This value should be between 0.0 and 1.0 inclusive. wait_printer_factory: (optional, defaults to bqclient.wait_printer_factory) Returns a subclass of WaitPrinter that will be called after each job poll. max_single_wait: Optional. Maximum number of seconds to wait for each call to query() / getQueryResults(). external_table_definitions_json: Json representation of external table definitions. udf_resources: Array of inline and remote UDF resources. location: Optional. The geographic location where the job should run. connection_properties: Optional. Connection properties to use when running the query, presented as a list of key/value pairs. A key of "time_zone" indicates that the query will be run with the default timezone corresponding to the value. job_creation_mode: Optional. An option for job creation. The valid values are JOB_CREATION_REQUIRED and JOB_CREATION_OPTIONAL. reservation_id: Optional. An option to set the reservation to use when execute the job. Reservation should be in the format of "project_id:reservation_id", "project_id:location.reservation_id", or "reservation_id". job_timeout_ms: Optional. How long to let the job run. max_slots: Optional. Cap on target rate of slot consumption by the query. **kwds: Passed directly to ExecuteSyncQuery. Raises: bq_error.BigqueryClientError: if no query is provided. StopIteration: if the query does not complete within wait seconds. bq_error.BigqueryError: if query fails. Returns: A tuple (schema fields, row results, execution metadata). For regular queries, the execution metadata dict contains the 'State' and 'status' elements that would be in a job result after FormatJobInfo(). For dry run queries schema and rows are empty, the execution metadata dict contains statistics z7Running RPC-style query asynchronously is not supportedNo query string providedNzrequest_id is not yet supportedriz#Wait timed out. Query not finished.rrrrrrrrrrr6rrrrrrtotalBytesProcessed)r)r) statisticscacheHitr schemarRUNNING)r6rr jobComplete)rrr'errorsSUCCESS)Staterrz Transient error during query: %sr)rrBigqueryClientErrorr rrrrintminsysmaxsizerr$rr\rrrrrrrr+rrrr)"rrrrrrrr6rrrrrrrrrrrrr3rr elapsed_timercurrent_wait_msremaining_time rows_to_readr_ executionr rowsrrvs" r* RunQueryRpcrpsb   & &A    & &'A BBEKK$I$I  & &'H II"$G++-Gyy{*,-/ \='/QTYY[:5Mll*n  $c./BTIJ Q KK/ ! ABB  #  ( ( 4  !#88Lx<OPL   * "  "6     "6 *   % -L (  #8 0! "*% ( *01F*G )6 !;A*;MIl #G ,Z 8  9?9IIl #G ,X 6b)$ $ V #%55BBII ~& -  m))<C    #&    - #0D b $   v #H-& ) i((% K p  . .= oo8!<<  ( (= oo8!<<=s, D(I2 B%I22KJ  K6KKdestination_tablepriorityallow_large_resultsflatten_resultsmaximum_billing_tierschema_update_optionslabelsc  |stjdd|i}!|jr"tj|j|!d<| r| |!d<| r| |!d<|r& tj ||}"t|"|!d <|r||!d <|r||!d <|r|j|!d <tj|!fid|d|d|d|d|d| d| d|d|d|d|d|d|d|d|d| d|d|d |tj|!|!d|!i}$t||d"#}%tj|$||||%$tj|$|%t||$fi| S#tj $r }#tj d|d|#d }#~#wwxYw)&aExecute the given query, returning the created job. The job will execute synchronously if sync=True is provided as an argument or if bqclient.sync is true. Args: bqclient: A BigqueryClient to get state and request clients from. query: Query to execute. destination_table: (default None) If provided, send the results to the given table. create_disposition: Optional. Specifies the create_disposition for the destination_table. write_disposition: Optional. Specifies the write_disposition for the destination_table. priority: Optional. Priority to run the query with. Either 'INTERACTIVE' (default) or 'BATCH'. preserve_nulls: Optional. Indicates whether to preserve nulls in input data. Temporary flag; will be removed in a future version. allow_large_results: Enables larger destination table sizes. dry_run: Optional. Indicates whether the query will only be validated and return processing statistics instead of actually running. use_cache: Optional. Whether to use the query cache. If create_disposition is CREATE_NEVER, will only run the query if the result is already cached. Caching is best-effort only and you should not make assumptions about whether or how long a query result will be cached. min_completion_ratio: Optional. Specifies the minimum fraction of data that must be scanned before a query returns. This value should be between 0.0 and 1.0 inclusive. flatten_results: Whether to flatten nested and repeated fields in the result schema. If not set, the default behavior is to flatten. external_table_definitions_json: Json representation of external table definitions. udf_resources: Array of inline and remote UDF resources. maximum_billing_tier: Upper limit for billing tier. maximum_bytes_billed: Upper limit for bytes billed. use_legacy_sql: The choice of using Legacy SQL for the query is optional. If not specified, the server will automatically determine the dialect based on query information, such as dialect prefixes. If no prefixes are found, it will default to Legacy SQL. schema_update_options: schema update options when appending to the destination table or truncating a table partition. labels: an optional dict of labels to set on the query job. query_parameters: parameter values for use_legacy_sql=False queries. time_partitioning: Optional. Provides time based partitioning specification for the destination table. clustering: Optional. Provides clustering specification for the destination table. destination_encryption_configuration: Optional. Allows user to encrypt the table created from a query job with a Cloud KMS key. range_partitioning: Optional. Provides range partitioning specification for the destination table. script_options: Optional. Options controlling script execution. job_timeout_ms: Optional. How long to let the job run. continuous: Optional. Whether the query should be executed as continuous query. job_creation_mode: Optional. An option for job creation. The valid values are JOB_CREATION_REQUIRED and JOB_CREATION_OPTIONAL. reservation_id: Optional. An option to set the reservation to use when execute the job. Reservation should be in the format of "project_id:reservation_id", "project_id:location.reservation_id", or "reservation_id". If reservation_id is "none", the job will be executed without assigned reservation using the on-demand slots. **kwds: Passed on to ExecuteJob. Raises: bq_error.BigqueryClientError: if no query is provided. Returns: The resulting job info. rrrrr)rA identifierzInvalid value z for destination_table: Nrjrl scriptOptionsjobCreationModerrcrrrdrr r!rrr"rtime_partitioning clusteringrr continuousrrange_partitioningrF)check_reservation_project)rr#rr)r) rrrrNrGetTableReference BigqueryErrorr\namerrWrrq)&rrrrcrdrrrrrrr rrr!rrr"r#rr(rr)r+script_optionsrrrrr*rrr3 query_configr5rvr]rs& r*rr4sMX   & &'A BB5!, %4%K%K&L!"%'FL#$3@L/0!33,=i(,IL#$),56$2L!&7&<&:.` to the fully qualified reservation path `projects//locations//reservations/`. The special value "none" is returned as is. Args: bqclient: A BigqueryClient to get state and request clients from. reservation_id: The reservation id to convert. check_reservation_project: Whether to validate the reservation project. Returns: The fully qualified reservation path or "none" if reservation_id is "none". nonerAr%default_locationr,)rNGetReservationReferencebq_flagsLOCATIONvaluepath)rrr,r5s r*rrsN"~7 55((.. 9 )  r,destination_table_referencesourcefield_delimiterskip_leading_rowsencodingquotemax_bad_recordsallow_quoted_newlines source_formatallow_jagged_rows!preserve_ascii_control_charactersignore_unknown_valuesprojection_fields autodetect null_marker null_markersuse_avro_logical_typesjson_extensioncopy_files_onlysource_column_matchc+ htj|tjjdt |i},t j |}-|-djt jr|-|,d<d}.n|-d}.|dtj|i|,d<|||,d<|||,d<|$|$|,d <|||,d <|||,d <|&|&|,d <||,d <|r||,d<|||,d<| | |,d<|!|!|,d<|"|"|,d<|#|#|,d<|*|*|,d<t j|,fid|d|d|d|d|d| d| d| d| d| d|d |d!|d"|d#|d$|d%|d&|d'|d(|d)|d*|%d+|'d,|)d-|&d.|,i}/|(Dtj||(tjjd/0}0|0j!|/d1<t#|f|/|.d2|+S)3aLoad the given data into BigQuery. The job will execute synchronously if sync=True is provided as an argument or if bqclient.sync is true. Args: bqclient: A BigqueryClient to get state and request clients from. destination_table_reference: TableReference to load data into. source: String specifying source data to load. schema: (default None) Schema of the created table. (Can be left blank for append operations.) create_disposition: Optional. Specifies the create_disposition for the destination_table_reference. write_disposition: Optional. Specifies the write_disposition for the destination_table_reference. field_delimiter: Optional. Specifies the single byte field delimiter. skip_leading_rows: Optional. Number of rows of initial data to skip. encoding: Optional. Specifies character encoding of the input data. May be "UTF-8" or "ISO-8859-1". Defaults to UTF-8 if not specified. quote: Optional. Quote character to use. Default is '"'. Note that quoting is done on the raw binary data before encoding is applied. max_bad_records: Optional. Maximum number of bad records that should be ignored before the entire job is aborted. Only supported for CSV and NEWLINE_DELIMITED_JSON file formats. allow_quoted_newlines: Optional. Whether to allow quoted newlines in CSV import data. source_format: Optional. Format of source data. May be "CSV", "DATASTORE_BACKUP", or "NEWLINE_DELIMITED_JSON". allow_jagged_rows: Optional. Whether to allow missing trailing optional columns in CSV import data. preserve_ascii_control_characters: Optional. Whether to preserve embedded Ascii Control characters in CSV import data. ignore_unknown_values: Optional. Whether to allow extra, unrecognized values in CSV or JSON data. projection_fields: Optional. If sourceFormat is set to "DATASTORE_BACKUP", indicates which entity properties to load into BigQuery from a Cloud Datastore backup. autodetect: Optional. If true, then we automatically infer the schema and options of the source files if they are CSV or JSON formats. schema_update_options: schema update options when appending to the destination table or truncating a table partition. null_marker: Optional. String that will be interpreted as a NULL value. null_markers: Optional. List of strings that will be interpreted as a NULL value. time_partitioning: Optional. Provides time based partitioning specification for the destination table. clustering: Optional. Provides clustering specification for the destination table. destination_encryption_configuration: Optional. Allows user to encrypt the table created from a load job with Cloud KMS key. use_avro_logical_types: Optional. Allows user to override default behaviour for Avro logical types. If this is set, Avro fields with logical types will be interpreted into their corresponding types (ie. TIMESTAMP), instead of only using their raw types (ie. INTEGER). reference_file_schema_uri: Optional. Allows user to provide a reference file with the reader schema, enabled for the format: AVRO, PARQUET, ORC. range_partitioning: Optional. Provides range partitioning specification for the destination table. hive_partitioning_options: (experimental) Options for configuring hive is picked if it is in the specified list and if it supports the precision and the scale. STRING supports all precision and scale values. If none of the listed types supports the precision and the scale, the type supporting the widest range in the specified list is picked, and if a value exceeds the supported range when reading the data, an error will be returned. This field cannot contain duplicate types. The order of the decimal_target_types: (experimental) Defines the list of possible SQL data types to which the source decimal values are converted. This list and the precision and the scale parameters of the decimal field determine the target type. In the order of NUMERIC, BIGNUMERIC, and STRING, a type is picked if it is in the specified list and if it supports the precision and the scale. STRING supports all precision and scale values. If none of the listed types supports the precision and the scale, the type supporting the widest range in the specified list is picked, and if a value exceeds the supported range when reading the data, an error will be returned. This field cannot contain duplicate types. The order of the types in this field is ignored. For example, ["BIGNUMERIC", "NUMERIC"] is the same as ["NUMERIC", "BIGNUMERIC"] and NUMERIC always takes precedence over BIGNUMERIC. Defaults to ["NUMERIC", "STRING"] for ORC and ["NUMERIC"] for the other file formats. json_extension: (experimental) Specify alternative parsing for JSON source format. To load newline-delimited JSON, specify 'GEOJSON'. Only applicable if `source_format` is 'NEWLINE_DELIMITED_JSON'. column_name_character_map: Indicates the character map used for column names. Specify 'STRICT' to use the latest character map and reject invalid column names. Specify 'V1' to support alphanumeric + underscore and name must start with a letter or underscore. Invalid column names will be normalized. Specify 'V2' to support flexible column name. Invalid column names will be normalized. file_set_spec_type: Set how to discover files for loading. Specify 'FILE_SYSTEM_MATCH' (default behavior) to expand source URIs by listing files from the underlying object store. Specify 'NEW_LINE_DELIMITED_MANIFEST' to parse the URIs as new line delimited manifest files, where each line contains a URI (No wild-card URIs are supported). thrift_options: (experimental) Options for configuring Apache Thrift load, which is required if `source_format` is 'THRIFT'. parquet_options: Options for configuring parquet files load, only applicable if `source_format` is 'PARQUET'. connection_properties: Optional. ConnectionProperties for load job. reservation_id: Optional. An option to set the reservation to use when execute the job. Reservation should be in the format of "project_id:reservation_id", "project_id:location.reservation_id", or "reservation_id". copy_files_only: Optional. True to configures the load job to only copy files to the destination BigLake managed table, without reading file content and writing them to new files. source_column_match: Optional. Controls the strategy used to match loaded columns to the schema. **kwds: Passed on to ExecuteJob. Returns: The resulting job info. rjr sourceUrisNfieldsr useAvroLogicalTypesreference_file_schema_urifileSetSpecType jsonExtensioncolumnNameCharacterMapparquetOptionsdecimalTargetTypesrltimeZone dateFormatdatetimeFormat timeFormattimestampFormatsourceColumnMatchrcrdr=r>r?r@rArCrBrDrErFrGr"rIrJr(r)rHr+hive_partitioning_optionsthrift_optionsrrMparquet_optionsloadFr4r)rr)rrPrrpr\rProcessSources startswithGCS_SCHEME_PREFIXrN ReadSchemarWr6r7r8r9r:rq)1rr;r<r rcrdr=r>r?r@rArBrCrDrErFrGrHr"rIrJr(r)rrKrSr+r_decimal_target_typesrLcolumn_name_character_map time_zone date_formatdatetime_format time_formattimestamp_formatfile_set_spec_typer`rarrrMrNr3 load_configsourcesrrr5s1 r*Loadrqs~!;#>#>#M#M$T*E%FG+  - -f 5' QZ-??@ 'K K!*K %'A'A&'IJK')?K%&*/HK+,#%7K!"#1K *,EK() $3K !&:+"#),45'K  +K  $3K ! +K !%5K!"$':K#$$$+*&  *    &"2*)J2*2 !" #$*%&'()*,+,!:-.$/0212&34&58;'-77!!**00"' I $->>#3M-  + HL r,destination_uris print_headerdestination_format compressionc xtj|tjjtjjfd|j d} | D]T} | j tjr#tjdj| tji}t|tjjrdt|i}n_t|tjjr;dt|i}|r|jdd|ii|r|jdd |iitj|| ||||| d |i}| Dt!j"|| t$j&j(d }|j+|d<t-|fd|i| S)aExtract the given table from BigQuery. The job will execute synchronously if sync=True is provided as an argument or if bqclient.sync is true. Args: bqclient: A BigqueryClient to get state and request clients from. reference: TableReference to read data from. destination_uris: String specifying one or more destination locations, separated by commas. print_header: Optional. Whether to print out a header row in the results. field_delimiter: Optional. Specifies the single byte field delimiter. destination_format: Optional. Format to extract table to. May be "CSV", "AVRO" or "NEWLINE_DELIMITED_JSON". trial_id: Optional. 1-based ID of the trial to be exported from a hyperparameter tuning model. add_serving_default_signature: Optional. Whether to add serving_default signature for BigQuery ML trained tf based models. compression: Optional. The compression type to use for exported files. Possible values include "GZIP" and "NONE". The default value is NONE. use_avro_logical_types: Optional. Whether to use avro logical types for applicable column types on extract jobs. reservation_id: Optional. An option to set the reservation to use when execute the job. Reservation should be in the format of "project_id:reservation_id", "project_id:location.reservation_id", or "reservation_id". **kwds: Passed on to ExecuteJob. Returns: The resulting job info. Raises: bq_error.BigqueryClientError: if required parameters are invalid. ExtractrB,z2Illegal URI: {}. Extract URI must start with "{}". sourceTable sourceModelmodelExtractOptionstrialIdaddServingDefaultSignature)rrrtrsr=rurKextractFr4rr)rrPrrpModelReferencesplitrdrrerrformatrTr\updaterWrNr6r7r8r9r:rq)rr5rrrsr=rttrial_idadd_serving_default_signaturerurKrr3urisuriextract_configrs r*rwrws`  % % 4 4  % % 4 4     $$ c >>,>> ?  ( ( > E E%77  . ;66EEF#T)_5N)[88GGH#T)_5N2Y4IJK$ *,I"  $$+%3n--77!!**00"' I $->>#3M- H BM BT BBr,r.)NNNNNNNN)NNFNCOPYN)F)NNNN)NNNNNNNNNNNNNNNNNNNNNN)NNNNN)NNN)rr)NNNNNNNNNNNNNNNNNNNNNNNNNNNNNN)T)(NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN)C__doc__ __future__rrrrrrrrtypingrrrr r r rabslr googleapiclientr rr7clientsrrr!rrNrrrrBigqueryClientrUrr+r4rrQboolr2r?rprirryr}rfloatJobCreationModerrrrqrr WaitPrinterrrrrrrrqrwrr,r*rsx *&% == 0#3, $ ,,,,38n,},sm ,^/88IM!% $48'+'+ $#'+,,+ 33DDE+#+ + 5cC01 +  } + }+~+C=+`IM!% $48'+'+ $#'R,,R 33DDER#R R 5cC01 R  } R }R~RC=Rr)-'+,1!$*15F ,,F K77FFGF  //>>F ! F  } F $D> F SMF "*#F R ,, **77   "(- ,, **77 tn H!%!% " I,,I I# I SM I sm I^# $%) $*.!% $$(#,0 $$(%)" $(%)")-5R,,R Rd^R~ R TN R  R#3-R#R RSMR}R#5/R R TN!R"sm#R& &&66'R,SM-R.TN/Rn! $!% $" ,E,,,E SM,E ,E# ,E  ,E sm ,Ed!%!% " '1,,'1 '1# '1 SM '1 sm '1Z $!% "  ,,   4.    #  SM  sm  J!% " ),,) ) SM)sm )^  P ,,P ..;;P  P   P #\---. P l ",,"..;;" "  "J,,B# $%) $*.!% ,0 %)$(" $($(#1A=,,A= A=d^A=~ A= TN A=  A=#3-A=#A= A=#5/A=#\---.A=c]A="sm#A=& &&66'A=,SM-A=.SM/A=0}1A=N(,(,'+"%)*." $,0&*$(*.*.%)15'+)-$(#%) $(Es/,,s/ s/ }s/! s/  } s/ sm s/TNs/"$s/d^s/~s/#5/s/d^s/#3-s/ #3-!s/"TN#s/$$DI.%s/& T#s(^ $'s/4SM5s/6}7s/8TN9s/> &&66?s/DSMEs/r'+,,SM $c] @ (,'+%)(,"%),0#'(,8<,0-1!%15!%(,)--1""$("$(&*)-Wr,,r!,!rC=r ~r(0~r $D>!r" S *#r$%r&$DI.'r(#)r*49%+r2%TN3r<SM=rRSMSrTd^UrV"#Wrv$(%)(, "&!%-1$(_C,,_C**99_C_C4. _C c] _C ! _C#_C%TN_CSM_Cr,