OdZddlZddlmZmZmZmZddlmZddl m Z ddl m Z ddl mZddl mZddl mZd Zd.d ej$fd Z d/d ej$d ed deefgdeej*j,deedeedeedeefdZ d/d ej$d ed deefgdeej*j,deedeedeedeefdZdZd ej$fdZd ej$dddefdZd ej$dddeefdZ d0d ej$fdZ d1d ej$dddeedeedeeeefd eeed!eed"eed#ee j@fd$Z! d.d ej$d%eefd&Z" d2d ej$d%eed#ee j@fd'Z# d3d ej$dej*jHd(ed)eeddf d*Z% d.d ej$d+ej*jHd,eejdefd-Z&y)4z(The BigQuery CLI dataset client library.N)DictList NamedTupleOptional) discovery)utils)bq_error) bq_id_utils)bq_processor_utilsexternalCatalogDatasetOptions apiclientct|}tj|d<|||d<|jjdi|j S)z(Get dataset with dataset_view parameter.accessPolicyVersion datasetView)dictbq_client_utils MAX_SUPPORTED_IAM_POLICY_VERSIONdatasetsgetexecute)r reference dataset_viewrequests %platform/bq/clients/client_dataset.py GetDatasetrsY O'66  )GM !    ! ! ,G , 4 4 66 id_fallbacksIDS project_idr max_results page_tokenlist_allfilter_expressionc *t|||||||dS)1List the datasets associated with this reference.r)#ListDatasetsWithTokenAndUnreachable)r rrr!r"r#r$s r ListDatasetsr(s0 -  rcvtj||}tj|tjj dt j||||}|||d<|jjd i|j}|jdg} t|jdg} |jdd} |d|vrt| |kr|t| z |d <|d|d <|jjd i|j}| j|jdg| j|jdg|jdd} d|vrt| |krt!| } | r| | d <| rt| | d<| S)r&)rrr(methodNallr unreachable nextPageToken maxResults pageToken)rtokenr)rNormalizeProjectReferencer typecheckApiClientHelperProjectReferencer PrepareListRequestrlistrrsetlenextendupdater) r rrr!r"r#r$rresult dataset_listunreachable_set next_tokenresponses rr'r'5s779)!!22   1 1j*; 'GEN $9    $ $ /w / 7 7 9&J+, ="56/zz/40* V #L(9K(G)C ,==gl#O4gk(y!!#((373;;=f&**Z45VZZ r:;::ot4j V #L(9K(G< (("HW"?3H] /rc tj|tjjdd|jd|j }ddt jii}|jj||jS)aFGets IAM policy for the given dataset resource. Arguments: apiclient: the apiclient used to make the request. reference: the DatasetReference for the dataset resource. Returns: The IAM policy attached to the given dataset resource. Raises: BigqueryTypeError: if reference is not a DatasetReference. GetDatasetIAMPolicyr* projects/ /datasets/optionsrequestedPolicyVersion)resourcebody) r r3r4DatasetReference projectId datasetIdrrr getIamPolicyr)r rformatted_resourcerHs rrBrBes!!22 "   ">> $|%wy rctj|tjjdd|jd|j }d|i}|j j||jS)a|Sets IAM policy for the given dataset resource. Arguments: apiclient: the apiclient used to make the request. reference: the DatasetReference for the dataset resource. policy: The policy string in JSON format. Returns: The updated IAM policy attached to the given dataset resource. Raises: BigqueryTypeError: if reference is not a DatasetReference. SetDatasetIAMPolicyr*rCrDpolicy)rHrG) r r3r4rIrJrKr setIamPolicyr)r rrPrMrs rrOrOsv!!22 " v '|+=|>wyrz,bq_id_utils.ApiClientHelper.DatasetReferencereturnctj|tjjd |j j dit |jy#tj$rYywxYw)z!Returns true if a dataset exists. DatasetExistsr*TFr r r3r4rIrrrrr BigqueryNotFoundErrorr rs rrTrTsp !!22  I/tI/779   ' ' s7A**B?Bc tj|tjjd |j j dit |jdS#tj$rYywxYw)z,Returns the region of a dataset as a string.GetDatasetRegionr*locationNrrUrWs rrYrYsu !!22   #9    # # 6d9o 6 > > @ LL  ' ' s9A,,BBcLtj|tjjdt j |}|||d<|||d<|||d<|||d<|||d<| d | i|d <|||d <| r%i|d <| j D] \}}||d |<| Mtj| tjjdd t j | di|d<| r| | d|d<|tj||t<|||d<|||d<|||d<t|j}tj|d< |jjdd|i|j!y#t"j$$r|sYywxYw)aCreate a dataset corresponding to DatasetReference. Args: apiclient: The apiclient used to make the request. reference: The DatasetReference to create. ignore_existing: (boolean, default False) If False, raise an exception if the dataset already exists. description: An optional dataset description. display_name: An optional friendly name for the dataset. acl: An optional ACL for the dataset, as a list of dicts. default_table_expiration_ms: Default expiration time to apply to new tables in this dataset. default_partition_expiration_ms: Default partition expiration time to apply to new partitioned tables in this dataset. data_location: Location where the data in this dataset should be stored. Must be either 'EU' or 'US'. If specified, the project that owns the dataset must be enabled for data location. labels: An optional dict of labels. default_kms_key: An optional kms dey that will apply to all newly created tables in the dataset, if no explicit key is supplied in the creating request. source_dataset_reference: An optional ApiClientHelper.DatasetReference that will be the source of this linked dataset. # external_source: External source that backs this dataset. connection_id: Connection used for accessing the external_source. external_catalog_dataset_options: An optional JSON string or file path containing the external catalog dataset options to create. max_time_travel_hours: Optional. Define the max time travel in hours. The value can be from 48 to 168 hours (2 to 7 days). The default value is 168 hours if this is not set. storage_billing_model: Optional. Sets the storage billing model for the dataset. resource_tags: An optional dict of tags to attach to the dataset. Raises: BigqueryTypeError: If reference is not an ApiClientHelper.DatasetReference or if source_dataset_reference is provided but is not an bq_id_utils.ApiClientHelper.DatasetReference. or if both external_dataset_reference and source_dataset_reference are provided or if not all required arguments for external database is provided. BigqueryDuplicateError: if reference exists and ignore_existing is False. CreateDatasetr*N friendlyName descriptionaccessdefaultTableExpirationMsdefaultPartitionExpirationMs kmsKeyNamedefaultEncryptionConfigurationrZlabels sourceDatasetdatasetReferencelinkedDatasetSource)externalSource connectionexternalDatasetReferencemaxTimeTravelHoursstorageBillingModel resourceTagsrrHr)r r3r4rIr ConstructObjectInfoitemsfrontend_utilsGetJson+EXTERNAL_CATALOG_DATASET_OPTIONS_FIELD_NAMErGetProjectReferencerrrinsertrr BigqueryDuplicateError)r rignore_existingr^ display_nameacldefault_table_expiration_msdefault_partition_expiration_ms data_locationrddefault_kms_keysource_dataset_referenceexternal_source connection_id external_catalog_dataset_optionsmax_time_travel_hoursstorage_billing_model resource_tagsrH label_key label_valueargss rr\r\s@!!22   / / :$'D%D_DN ,'BD #$$0+JD '( .:O-LD )*$D DN"(,,. ;"-d8nY#1) ##44 +?? $ #D )#(D #$&18F8N8N(9D 45&!6D &"7D (D i++- .$ / P P$ I2T2T2::<  ( (    s0F F#"F#r^rwtags_to_attachtags_to_removeclear_all_tagsr update_modectj|tjjdt ||| }|||d<|||d<|||d<|||d<||dk(rd|d <n||d <| d | i|d <d |vri|d <|r |j D] \}}||d |<|r|D] }d|d |< | | |d <| | |d<i}|rd|vr|dD]}d||< n|xsgD]}d||< | xsiD] }| |||< ||d<|<|j ti|t}tj|||t<t|||| |y)axUpdates a dataset. Args: apiclient: The apiclient used to make the request. reference: The DatasetReference to update. description: An optional dataset description. display_name: An optional friendly name for the dataset. acl: An optional ACL for the dataset, as a list of dicts. default_table_expiration_ms: Optional number of milliseconds for the default expiration duration for new tables created in this dataset. default_partition_expiration_ms: Optional number of milliseconds for the default partition expiration duration for new partitioned tables created in this dataset. labels_to_set: An optional dict of labels to set on this dataset. label_keys_to_remove: An optional list of label keys to remove from this dataset. etag: If set, checks that etag in the existing dataset matches. default_kms_key: An optional kms dey that will apply to all newly created tables in the dataset, if no explicit key is supplied in the creating request. max_time_travel_hours: Optional. Define the max time travel in hours. The value can be from 48 to 168 hours (2 to 7 days). The default value is 168 hours if this is not set. storage_billing_model: Optional. Sets the storage billing model for the dataset. tags_to_attach: An optional dict of tags to attach to the dataset tags_to_remove: An optional list of tag keys to remove from the dataset clear_all_tags: If set, clears all the tags attached to the dataset external_catalog_dataset_options: An optional JSON string or file path containing the external catalog dataset options to update. update_mode: An optional flag indicating which datasets fields to update, either metadata fields only, ACL fields only, or both metadata and ACL fields. Raises: BigqueryTypeError: If reference is not a DatasetReference. UpdateDatasetr*Nr]r^r_r`rrarbrcrdrkrlrm) r r3r4rI_ExecuteGetDatasetRequestro setdefaultrrrp#UpdateExternalCatalogDatasetOptions_ExecutePatchDatasetRequest)r rr^rwrxryrz labels_to_setlabel_keys_to_removeetagr|rrrrrrrdatasetrrrtagcurrent_optionss rrrOsr!!22  &iD A'*GN(GM_GH ,*EG &'$0&!+04g,-0Og,- 1=0OG ,- WGH"/"5"5"7 ;%0gh "#8) %)gh "*&$9G !&%:G !"-'1~&mC'##mC$  !r !c',M#"*'.%1 BBGIJO:: =  78    rrct|}tj|d<|jjdi|}|r||j d<|j }|S)aExecutes request to get dataset. Args: apiclient: the apiclient used to make the request. reference: the DatasetReference to get. etag: if set, checks that etag in the existing dataset matches. Returns: The result of executing the request, if it succeeds. rIf-Matchr)rrrrrheadersr)r rrr get_requestrs rrrsb i$ / P P$( ""$((040+ &*K #    !' .rct|}tj|d<||j|d<|j j dd|i|}|s|dr|r|n|d|j d<|jy)a<Executes request to patch dataset. Args: apiclient: the apiclient used to make the request. reference: the DatasetReference to patch. dataset: the body of request etag: if set, checks that etag in the existing dataset matches. update_mode: a flag indicating which datasets fields to update. rN updateModerHrrr)rrrvaluerpatchrr)r rrrr parametersrs rrrs I*66 "#*00J| &I   & & BG Bz B' WV_*.$GFOGOOJ //rignore_not_founddelete_contentsctj|tjjdt |}|||d< |j j di|jy#tj$r|sYywxYw)aYDeletes DatasetReference reference. Args: apiclient: the api client to make the request with. reference: the DatasetReference to delete. ignore_not_found: Whether to ignore "not found" errors. delete_contents: [Boolean] Whether to delete the contents of non-empty datasets. If not specified and the dataset has tables in it, the delete will fail. If not specified, the server default applies. Raises: BigqueryTypeError: if reference is not a DatasetReference. bq_error.BigqueryNotFoundError: if reference does not exist and ignore_not_found is False. DeleteDatasetr*NdeleteContentsr) r r3r4rIrrdeleterr rV)r rrrrs rrrs*!!22  i$ ,D  I'$'//1  ' '    s.A33B  B dataset_reference timestampc t|}|r*dtj|jddi|d<|j j di|j S#tj$r}|d}~wwxYw)aUndeletes a dataset. Args: apiclient: The api client to make the request with. dataset_reference: [Type: bq_id_utils.ApiClientHelper.DatasetReference]DatasetReference of the dataset to be undeleted timestamp: [Type: Optional[datetime.datetime]]Timestamp for which dataset version is to be undeleted Returns: bool: The job description, or None for ignored errors. Raises: BigqueryDuplicateError: when the dataset to be undeleted already exists. deletionTimez+00:00rHNr) rrp FormatRfc3339replacerundeleterr ru)r rrres rUndeleteDatasetr.s*  ! "D .66yAII d6l )9    ( ( 04 0 8 8 ::  ( ( G sA$A''B:A<<B)N)NNNNN)FNNNNNNNNNNNNNNN)NNNNNNNNNNNNNFNN)NN)FN)'__doc__datetimetypingrrrrgoogleapiclientrclientsrrfrontendrpr r r rrResourcerstrr4r5intboolr(r'rBrOrTrYr\ UpdateModerrrrIrrrrrrs.33%,,$.M+7),,7&IM!% $#'+!! 8C= )  33DDE# tn }DIM!% $#'+-!!- 8C= ) - 33DDE-#- -tn- }-`$N9#5#5>!!= " !! = c] * $$( !%)%~ !!~ H"&"& $$( /3*.%*6:8<%z!!z=z#z3- zT#s(^,zT#Y'z TN!z"'/sm#z$/445%z@!! 3-88< !!!! 3- ! /445 !N#&* # !!# **;;# # d^ #  # R.2  !!  "22CC  ))*    r