~dZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZddlm Z dd lmZdd lmZddlmZddlmZddlZdZdZdZdZ dZ!dZ"d#dZ#GddZ$GddZ%GddZ&GddZ'Gdd Z(Gd!d"Z)y)$z4Helpers for interacting with the Cloud Dataflow API.)absolute_import)division)unicode_literalsN)encoding) exceptions)cloudbuild_util) storage_api) storage_util)apis) submit_util)log) properties)filesdataflowv1b3z us-central1c>tjttSN)r GetMessagesModuleDATAFLOW_API_NAMEDATAFLOW_API_VERSION+lib/googlecloudsdk/api_lib/dataflow/apis.pyrr.   13G HHrc>tjttSr)r GetClientInstancerrrrrrr2rrcjtjjjj dS)NT)required)rVALUEScoreprojectGetrrr GetProjectr#6s(      ' ' + +T + ::rc|sd|dSd|dS)z/Returns full the image path of the given image.zgcr.io/dataflow-templates-base/z-template-launcher-base:latestz)-template-launcher-base-distroless:latestr)image is_distrolesss r_GetBaseImagePathr':s)  )%0NO +5'1Z [[rc,eZdZdZej ZejZejZ ejZ e dZe d dZe d dZe d dZe d dZe d dZe dd Zy)Jobsz'The Jobs set of Dataflow API functions.c*tjSrrprojects_locations_jobsrrr GetServicezJobs.GetServiceM   6 66rNc|xs t}|xst}tj||||} tj j |S#tj$r}tj|d}~wwxYw)aWCalls the Dataflow Jobs.Get method. Args: job_id: Identifies a single job. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. view: (DataflowProjectsJobsGetRequest.ViewValueValuesEnum) Level of information requested in response. Returns: (Job) )jobIdlocation projectIdviewN) r#DATAFLOW_API_DEFAULT_REGIONr'DataflowProjectsLocationsJobsGetRequestr)r-r"apitools_exceptions HttpErrorr HttpException)job_id project_id region_idr3requesterrors rr"zJobs.GetQs+z|J88I!IIyJTJKG, __  " "7 ++  ( (,  $ $U ++,"AB-BBcB|xs t}|xst}d}|rRtjj tjjj ddg}tj|tjj j}tj||||} tjj|S#tj$r}tj|d}~wwxYw)aCancels a job by calling the Jobs.Update method. Args: job_id: Identifies a single job. force: True to forcibly cancel the job. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. Returns: (Job) Nforce_cancel_jobtruekeyvalueadditionalProperties)labelsrequestedStater0r1r2job)r#r4rJob LabelsValueAdditionalPropertyRequestedStateValueValuesEnumJOB_STATE_CANCELLED*DataflowProjectsLocationsJobsUpdateRequestr)r-Updater6r7rr8)r9forcer:r;rGrJr<r=s rCancelz Jobs.Cancelis +z|J88I F  "&&22   ! ! - - @ @$F A 4I2 f   ! !)+//MM,, " /C !LLyJCMIG, __  % %g ..  ( (,  $ $U ++,s"C11DDDc"|xs t}|xst}tjtj |||rdn|}g}||j d||j d||r|j ddj |} tj||||| } tjj| S#tj$r} tj| d} ~ wwxYw) a0Update pipeline options on a running job. You should specify at-least one (or both) of min_num_workers and max_num_workers. Args: job_id: ID of job to update project_id: Project of the job region_id: Region the job is in min_num_workers: Lower-bound for worker autoscaling max_num_workers: Upper-bound for worker autoscaling worker_utilization_hint: Target CPU utilization for worker autoscaling unset_worker_utilization_hint: Unsets worker_utilization_hint value Returns: The updated Job N) minNumWorkers maxNumWorkersworkerUtilizationHint)runtimeUpdatableParamsz(runtime_updatable_params.min_num_workersz(runtime_updatable_params.max_num_workersz0runtime_updatable_params.worker_utilization_hint,)r0r1r2rJ updateMask)r#r4rrKRuntimeUpdatableParamsappendjoinrPr)r-rQr6r7rr8) r9r:r;min_num_workersmax_num_workersworker_utilization_hintunset_worker_utilization_hintrJupdate_mask_pieces update_maskr<r=s r UpdateOptionszJobs.UpdateOptionss&8+z|J88I   ! !02II))1, J  " C" JK" JK+ ( <((-.K!LL  MG, __  % %g ..  ( (,  $ $U ++,s>"C!!D4D  Dc|xs t}|xst}tjtjjj }tj ||||} tjj|S#tj$r}tj|d}~wwxYw)zDrains a job by calling the Jobs.Update method. Args: job_id: Identifies a single job. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. Returns: (Job) )rHrIN)r#r4rrKrNJOB_STATE_DRAINEDrPr)r-rQr6r7rr8)r9r:r;rJr<r=s rDrainz Jobs.Drains+z|J88I   ! !)+//MM** " -C !LLyJCMIG, __  % %g ..  ( (,  $ $U ++,s7"BC-CCc~|xs t}|xst}tj|g}tj |}tj ||||} t jj|S#tj$r}tj|d}~wwxYw)aYResumes a job by calling the Jobs.Update method. Args: job_id: Identifies a single job. experiment_with_token: The resume token unique to the job prefixed with the experiment key. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. Returns: (Job) ) experiments) environmentrIN) r#r4r EnvironmentrKrPr)r-rQr6r7rr8)r9experiment_with_tokenr:r;rjrJr<r=s rResumeUnsupportedSDKzJobs.ResumeUnsupportedSDKs"+z|J88I#%11*+2-K   ! !k ! :C!LLyJCMIG, __  % %g ..  ( (,  $ $U ++,s,"BB<"B77B<c H|xs t}|xst}tj|||tj |||} t j j|S#tj$r}tj|d}~wwxYw)abTakes a snapshot of a job via the Jobs.Snapshot method. Args: job_id: Identifies a single job. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. ttl: The ttl for the snapshot. snapshot_sources: If true, the sources will be snapshotted. Returns: (Snapshot) )r1ttlsnapshotSources)r0r1r2snapshotJobRequestN) r#r4r,DataflowProjectsLocationsJobsSnapshotRequestSnapshotJobRequestr)r-Snapshotr6r7rr8)r9r:r;rosnapshot_sourcesr<r=s rrtz Jobs.Snapshots$+z|J88I!NN,.AAC9IBK OG, __  ' ' 00  ( (,  $ $U ++,s"A44B!BB!NNN)FNN)NNNNNN)NN)NN604800sF)__name__ __module__ __qualname____doc__rr5 GET_REQUEST(DataflowProjectsLocationsJobsListRequest LIST_REQUEST%DataflowProjectsJobsAggregatedRequestAGGREGATED_LIST_REQUESTrPUPDATE_REQUEST staticmethodr-r"rSrdrgrmrtrrrr)r)Cs/!#KK+"$MM,-))$..77,,.,,@"$(A,A,F,,0'+%),,8 % ,,rr)cTeZdZdZej ZedZeddZ y)Metricsz*The Metrics set of Dataflow API functions.c*tjSrr+rrrr-zMetrics.GetService,r.rNc|xs t}|xst}tj||||} tj j |S#tj$r}tj|d}~wwxYw)aCalls the Dataflow Metrics.Get method. Args: job_id: The job to get messages for. project_id: The project which owns the job. region_id: The regional endpoint of the job. start_time: Return only metric data that has changed since this time. Default is to return all information about all metrics for the job. Returns: (MetricUpdate) )r0r1r2 startTimeN) r#r4r.DataflowProjectsLocationsJobsGetMetricsRequestrr- GetMetricsr6r7rr8)r9r:r; start_timer<r=s rr"z Metrics.Get0s+z|J88I44 5 ,    ! , ,W 55  ( (,  $ $U ++,r>rv) rxryrzr{rrr|rr-r"rrrrr&s@2!2277,,rrceZdZdZdZdZdZdZdZdZ dZ dZ dZ dZ dZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZ ddZy)TemplateArgumentsz%Wrapper class for template arguments.Ncp||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||_ ||_||_||_||_||_||_||_||_||_||_||_||_yr)r:r;job_name gcs_locationzone max_workers num_workersnetwork subnetworkworker_machine_typelauncher_machine_typestaging_location temp_location kms_key_namedisable_public_ips parametersservice_account_email worker_region worker_zoneenable_streaming_engineadditional_experimentsadditional_pipeline_optionsadditional_user_labelsstreaming_updatetransform_name_mappings flexrs_goal)selfr:r;rrrrrrrrrrrrrrrrrrrrrrrrs r__init__zTemplateArguments.__init__ks6!DODNDM$DDI"D"DDL DO2D!6D,D&D$D0D DO!6D&D"D#:D "8D'BD$"8D,D#:D "Dr)NNNNNNNNNNNNNNNNNNNNNNNNNN)rxryrzr{r:r;rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrMs-*), ( $++ '*-,*-+  $ + #'%) $! "&%)!'+&*+/&* $'+54#rrceZdZdZej ZejZejZ ejZ ejZ ejZej Zej$Zej(ZejZej.Zejj2Zejj6Zej:Zej>Z ej>jBZ"ejFZ$e%dZ&e%dZ'e%dZ(e%ddZ)e%ddZ*e%ddZ+e%dZ,e%dZ-e%ddZ.dZ/e0jbe0jdze0jfzZ4e0jje0jlze0jnze0jpzZ9e0jbe0jtze0jnze0jvze0jpzZe=d Z?e=d d Z@e=d d ZAe=dZBe=dZCe=dZDe=dZEe=dZFe=dZGe=dZHe=dZIe=dZJe=dZKe=dZLe=dZMe=dZNe= d!dZOe=dZPe=dZQe=dZRe=d dZSy )" Templatesz,The Templates set of Dataflow API functions.java11java17java21Tpython3goz4gs://dataflow-templates-{}/latest/flex/Yaml_Templatec*tjSr)rprojects_locations_templatesrrrr-zTemplates.GetService   ; ;;rc*tjSr)r projects_locations_flexTemplatesrrrGetFlexTemplateServicez Templates.GetFlexTemplateService   ? ??rNc$g}|j}|rtj|niD]5\}}|jtj j ||7|jxst}tjj}|j}|jr|nd}tjj} tj|j | } tj#|j$|j&|tj|j(|j*|j,|j.|j0|j2|j4|j6|j8||j:|j<|j>| r | | nd|j@r |j@ng|jBr |jBng|rtj |nd} tjE|jFxs tI|| } tjKjM| S#tNjP$r} tSjT| d} ~ wwxYw)zCalls the Dataflow Templates.CreateFromJob method. Args: template_args: Arguments for create template. Returns: (Job) rBNrE)serviceAccountEmailr maxWorkers numWorkersrr machineType tempLocation kmsKeyNameipConfiguration workerRegion workerZoneenableStreamingEngineadditionalUserLabelsadditionalExperimentsadditionalPipelineOptions)gcsPathjobNamer1rjr)r2r1createJobFromTemplateRequest)+rsix iteritemsr\rPARAMETERS_VALUErMr;r4rRuntimeEnvironmentIpConfigurationValueValuesEnumWORKER_IP_PRIVATErAdditionalUserLabelsValue _Templates__ConvertDictArgumentsrCREATE_REQUESTrrrrrrrrrrrrrrrr/DataflowProjectsLocationsTemplatesCreateRequestr:r#r-Creater6r7rr8) template_args params_listrkvr;ip_configuration_enum ip_privateip_configurationuser_labels_valueuser_labels_listbodyr<r=s rrzTemplates.CreatesnK))J-7 j)R?1  $ $ 7 7AQ 7 GI@ ''F+FI-77&88J%2%E%Ez4)22 77,,  # #**&&%':: - C C##$00$00!))$//%99&77$11,&44$00"/"G"G "3%5"7%)!7744!<<993;82<--!,..AEA $ GDB 55**:jl%)6+ ,  ! ! # * *7 33  ( (,  $ $U ++,s?"I""J5J  Jc&g}|j}|rtj|niD]5\}}|jtj j ||7tj|jtj}d}d}|jr$|j}|rtj|}|jxst}tjj} | j } |j"r| nd} tj%tj|j&|j(|j*|j,|j.|j0|j2|j4|j6| |j8|j:|j<|j>r |j>ng|j@|rtj |nd||} tjC|jD|| |jFxs tId} tjKjM| S#tNjP$r}tSjT|d}~wwxYw)aCalls the Dataflow Templates.LaunchTemplate method on a dynamic template. Args: template_args: Arguments to create template. gcs_location must point to a Json serialized DynamicTemplateFileSpec. Returns: (LaunchTemplateResponse) rBNrE)rrrrrrrrrrrrrr)rjrrupdatetransformNameMappingF)rr1launchTemplateParametersr2 validateOnly)+rrrr\r LAUNCH_TEMPLATE_PARAMETERS_VALUErMrr-DYNAMIC_TEMPLATE_TRANSFORM_NAME_MAPPING_VALUErr;r4rrrrrLAUNCH_TEMPLATE_PARAMETERSrrrrrrrrrrrrrr/DataflowProjectsLocationsTemplatesLaunchRequestrr:r#r-Launchr6r7rr8)rrrrrtransform_mapping_listtransform_mappingsrr;rrrrr<r=s rLaunchDynamicTemplatezTemplates.LaunchDynamicTemplate&svK))J-7 j)R?1  4 4 G G1 H @ '==--??%%&77   C C%; D  ''F+FI-77&88J%2%E%Ez4  / /%':: - C C##$00$00!))$//%99&77$11,&44$00"/"G"G!7744#; (&&1<==!,>.BF/3 0 D8 KK!..%)#..>*, L  ,  ! ! # * *7 33  ( (,  $ $U ++,s"I##J6J  Jcg}|r?tj|D]'\}}|j|j||)|S)zConvert dictionary arguments to parameter list . Args: arguments: Arguments for create job using template. value_message: the value message of the arguments Returns: List of value_message.AdditionalProperty rB)rrr\rM) arguments value_messagerrrs r__ConvertDictArgumentsz Templates.__ConvertDictArgumentsysIK-- *$!Q=;;;KL+ rcd}d}d|d<t|jDcgc]\}}dj||}}}dj|}dj|} dj| } t j |jt j||| dj| } | Scc}}w) aFBuilds Dockerfile contents for java flex template image. Args: flex_template_base_image: SDK version or base image to use. pipeline_paths: List of paths to pipelines and dependencies. env: Dictionary of env variables to set in the container image. Returns: Dockerfile contents as string. B FROM {base_image} {env} {copy} {commands} z /template/*FLEX_TEMPLATE_JAVA_CLASSPATH ENV {}={}  COPY {} /template/ base_imageenvcopycommands)sorteditemsformatr]textwrapdedentr_GetFlexTemplateBaseImage) flex_template_base_imagepipeline_pathsrdockerfile_templaterrrenvsenv_listpaths copy_commanddockerfile_contentss rBuildJavaImageDockerfilez"Templates.BuildJavaImageDockerfilesH*7C&'17 1D E1DAK  q! $1DD EyyH HH^ $E'..u5L"//*=>EE66 $&  8$ F&  FsCc  d}gd}d|d|d<d|vra|djd}|rK|Dcgc]}d| }}dj||d<dj|}|jd|d |vr3d|d |d <|jd |d |jd d |vr d|d |d <t|j D cgc]\} } d j | | } } } dj| } dj|} dj | }t j|j tj|| |ddj|z}|Scc}wcc} } w)aHBuilds Dockerfile contents for python flex template image. Args: flex_template_base_image: SDK version or base image to use. pipeline_paths: List of paths to pipelines and dependencies. env: Dictionary of env variables to set in the container image. Returns: Dockerfile contents as string. r)zapt-get updatez!apt-get install -y libffi-dev gitzrm -rf /var/lib/apt/lists/*z /template/FLEX_TEMPLATE_PYTHON_PY_FILE#FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGESrYrz pip install &FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILEz!pip install --no-cache-dir -U -r a(pip check || (e=$?; echo "Building a container with incompatible dependencies is prevented by default. If you are sure you want to proceed, you need to create your own container image. See: https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates"; exit $e))FLEX_TEMPLATE_PYTHON_SETUP_FILErrrzRUN z && r) splitr]r\rrrrrrr)rrrrr package_listpackage packages_path package_argrrrrrrrs rBuildPythonImageDockerfilez$Templates.BuildPythonImageDockerfilesH S789:&'-3>?EEcJl ?KL|G:gY/| L58XXm5L 12hh}- ;- ( 036 sCDE F 23oo :; < >oo )C/ s<=> ? +,28 1D E1DAK  q! $1DD EyyH HH^ $E'..u5L"//*=>EE66 $&  &++h// F1 KM4 Fs E:E?cd}dj|d|d<dj|}dj|}t|jDcgc]\}}dj||}}}dj|} t j |jt j|| |} | Scc}}w) a0Builds Dockerfile contents for go flex template image. Args: flex_template_base_image: SDK version or base image to use. pipeline_paths: Path to pipeline binary. env: Dictionary of env variables to set in the container image. Returns: Dockerfile contents as string. z2 FROM {base_image} {env} {copy} z /template/{}FLEX_TEMPLATE_GO_BINARYrrrr)rrr)rr]rrrrrr) rrrrrrvarvalrrrs rBuildGoImageDockerfilez Templates.BuildGoImageDockerfiles&4%:%: %&&(C!" HH^ $E'..u5L6EE66 $&   F  sCc|dk(rtj|||S|dk(rtj|||S|dk(rtj|||Sy)axBuilds Dockerfile contents for flex template image. Args: flex_template_base_image: SDK version or base image to use. pipeline_paths: List of paths to pipelines and dependencies. env: Dictionary of env variables to set in the container image. sdk_language: SDK language of the flex template. Returns: Dockerfile contents as string. JAVAPYTHONGON)rrrr)rrr sdk_languages rBuildDockerfilezTemplates.BuildDockerfile"sxv  / /0H0>EE  !  1 12J2@#GG    - -.F. @@ rctjtj|}tj}|jst dj ||j|_|jr|j|_|jr0tj|j|j|_|jr|j|_ |S)aBuilds and validates TemplateMetadata object. Args: template_metadata_json: Template metadata in json format. Returns: TemplateMetadata object on success. Raises: ValueError: If is any of the required field is not set. zEInvalid template metadata. Name field is empty. Template Metadata: {}) r JsonToMessagerTEMPLATE_METADATAr r!r descriptionrr%yamlDefinition)template_metadata_jsontemplate_metadatatemplate_metadata_objs r_BuildTemplateMetadataz Templates._BuildTemplateMetadataks!..y/J/J/EG%779  ! ! 0067H0I KK!2!7!7$$*;*G*G'##++,=,H,HI):)E)E&''->-M-M*  rc|jr |jS|jr |jS|jr tj |jStj tS#t j $rY.wxYw)+Returns the image path for a YAML template.)r% yaml_imagerr_ExtractYamlTemplateImagerr8r4)argss rGetYamlTemplateImagezTemplates.GetYamlTemplateImages{ zz ZZ  __  2243E3EFF  . ./J KK % %   sA66B  B cftjj|}tj}t j j|} tj|j|}|dS#t$r*}tjdj||d}~wwxYw)r3zSUnable to read file {0} due to incorrect file path or insufficient read permissionsNr%)rYAML_TEMPLATE_GCS_LOCATIONrr StorageClientr ObjectReferenceFromUrljsonload ReadObject Exceptionrr8)r;yaml_gcl_template_pathstorage_clientobj_refgeneric_template_definitiones rr5z#Templates._ExtractYamlTemplateImages'AAHH!..0N**223IJG$(II  # #G ,%! 'w //   $ $ $f%;< s$A== B0%B++B0c|dk(rtjS|dk(rtjS|dk(rtjS|dk(rtjS|dk(rtj S|dk(rtj S|dk(r%tjdtjS|d k(rtjS|d k(rtjS|d k(rtjS|S) aReturns latest base image for given sdk version. Args: flex_template_base_image: SDK version or base image to use. Returns: If a custom base image value is given, returns the same value. Else, returns the latest base image for the given sdk version. JAVA11JAVA17JAVA21JAVA11_DISTROLESSJAVA17_DISTROLESSJAVA21_DISTROLESSJAVA8z]JAVA8 is deprecated and redirected to JAVA11. This option will be removed in a future releasePYTHON3r GO_DISTROLESS) rFLEX_TEMPLATE_JAVA11_BASE_IMAGEFLEX_TEMPLATE_JAVA17_BASE_IMAGEFLEX_TEMPLATE_JAVA21_BASE_IMAGE*FLEX_TEMPLATE_JAVA11_DISTROLESS_BASE_IMAGE*FLEX_TEMPLATE_JAVA17_DISTROLESS_BASE_IMAGE*FLEX_TEMPLATE_JAVA21_DISTROLESS_BASE_IMAGEr warning FLEX_TEMPLATE_PYTHON3_BASE_IMAGEFLEX_TEMPLATE_GO_BASE_IMAGE&FLEX_TEMPLATE_GO_DISTROLESS_BASE_IMAGE)rs rrz#Templates._GetFlexTemplateBaseImages 8+  6 66 !X -  6 66 !X -  6 66 !%8 8  A AA !%8 8  A AA !%8 8  A AA !W , kk 0 6 66 !Y .  7 77 !T )  2 22 !_ 4  = == ##rc|dk(r.tjtjjS|dk(r.tjtjjS|dk(r.tjtjj S|dk(r.tjtjj Sy)zBuilds SDKInfo object. Args: sdk_language: SDK language of the flex template. Returns: SDKInfo object r)languagerYAMLrN)rSDK_INFO SDK_LANGUAGErrr\r)rs r _BuildSDKInfozTemplates._BuildSDKInfosv   )?)?)D)D  EE  !   )?)?)F)F  GG     )?)?)D)D  EE     )?)?)B)B  CC rcPtj5}tjj |d}tj ||dt j}tjj|}|j||cdddS#1swYyxYw)a>Stores flex template container spec file in GCS. Args: template_file_gcs_location: GCS location to store the template file. container_spec_json: Container spec in json format. Returns: Returns the stored flex template file gcs object on success. Propagates the error on failures. ztemplate-file.jsonr)newlineN) rTemporaryDirectoryospathr]WriteFileContentsr r:r r;r< CopyFileToGCS)template_file_gcs_locationcontainer_spec_jsontemp_dir local_pathrBrCs r_StoreFlexTemplateFilez Templates._StoreFlexTemplateFiles{  ! ! #x77<<*>?j j*=tL"002n,,445OPg  ) )*g > $ # #s A=BB%c d} |rtj|} tj|} d} |rBtj|jtj } tj j} |jr| nd}|jrdnd}tj|j|j|j|j|j|j |j"r |j"n |j$|j$|j&||j(|j*||j,r |j,ng| rtj | nd} tj/|| | | |||}t1j2|}t5j6t5j8|ddd}|r|S tj;||t<j>jAd jC||y#tDjF$r}tIjJ|d}~wwxYw) aBuilds container spec and stores it in the flex template file in GCS. Args: template_file_gcs_location: GCS location to store the template file. image: Path to the container image. template_metadata_json: Template metadata in json format. sdk_language: SDK language of the flex template. print_only: Only prints the container spec and skips write to GCS. template_args: Default runtime parameters specified by template authors. image_repository_username_secret_id: Secret manager secret id for username to authenticate to private registry. image_repository_password_secret_id: Secret manager secret id for password to authenticate to private registry. image_repository_cert_path: The full URL to self-signed certificate of private registry in Cloud Storage. Returns: Container spec json if print_only is set. A success message with template file GCS path and container spec otherewise. NTrE)rrrrrrrstagingLocationrrrrrrr)r%metadatasdkInfodefaultEnvironmentimageRepositoryUsernameSecretIdimageRepositoryPasswordSecretIdimageRepositoryCertPath)rYz: ) sort_keysindent separatorszkSuccessfully saved container spec in flex template file. Template File GCS Location: {} Container Spec: {})&rr1r_rrFLEX_TEMPLATE_USER_LABELS_VALUEIP_CONFIGURATION_ENUM_VALUErrrFLEX_TEMPLATE_ENVIRONMENTrrrrrrrrrrrrCONTAINER_SPECr MessageToJsonr=dumpsloadsrkr statusPrintrr6r7rr8)rgr%r.r print_onlyr#image_repository_username_secret_id#image_repository_password_secret_idimage_repository_cert_pathr/sdk_infodefault_environmentrrrrcontainer_specrhcontainer_spec_pretty_jsonr=s rBuildAndStoreFlexTemplateFilez'Templates.BuildAndStoreFlexTemplateFiles><#:: "&&|4H"99  . .  3 3588JJj%88*d 77$T&??+AA".."..''"--#77  ( (%22.;.L.L'88"//*$22".. 7!.!E!E"/!E!EKM )HH#3 I 5#''@)(--".(K(K :. >>KHHT;rctj|D]s\}}}tj|tj|D]D}tjtj j ||tjFuy)zWalks a dir to chmod itself and its contents with the configured access. Args: top_dir_path: The path to the top-level directory. Raises: OSError: If the chmod fails. N)rcwalkrrDIR_PERMISSIONS_MASKrdr]FILE_PERMISSIONS_MASK) top_dir_pathdirpath_ filenamesfilenames r_ChmodRWorldReadablezTemplates._ChmodRWorldReadableksf"$!6I)G)GH(!! GGLL( +Y-L-L  "7rc rtj||tj5} tj j dj| g} |} |r|} n|r|g} | D]} tjj| } tjj| r>) $ll9h7)   % %9:: __ggll8RWW%5%5i%@A (    * *8 4 bggmmI6q9:;> jj OQ%556N6Dc6BD X||j>|j@||jB|jD|jF| |jHr |jHng|rtj|nd|jJr |jJng|||rtj|nd} tMjO|jPxs tS|| } tjUjW| S#tXjZ$r} t]j^| d} ~ wwxYw) zCalls the create job from flex template APIs. Args: template_args: Arguments for create template. Returns: (Job) NrESPEED_OPTIMIZEDCOST_OPTIMIZED)rrrrrrlauncherMachineTyperrmrrrrr flexrsGoalrrr)rcontainerSpecGcsPathrjrtransformNameMappingsr)launchParameter)r2r1launchFlexTemplateRequest)0rrrFLEX_TEMPLATE_PARAMETERS_VALUEr*FLEX_TEMPLATE_TRANSFORM_NAME_MAPPING_VALUErrrxr;r4ryrrrFLEXRS_GOAL_ENUM_VALUEFLEXRS_SPEED_OPTIMIZEDFLEXRS_COST_OPTIMIZEDLAUNCH_FLEX_TEMPLATE_REQUESTFLEX_TEMPLATE_PARAMETERrrrzrrrrrrrrrrrrrrrr3DataflowProjectsLocationsFlexTemplatesLaunchRequestr:r#rrr6r7rr8) rrrrrrr;rrrrr<r=s rCreateJobFromFlexTemplatez#Templates.CreateJobFromFlexTemplates22  )"J"JLK&==--<<>%%&77   @ @%; A   !77,,113 ''F+FI66HHJ%2%E%Ez4K   " "&7 766MM  $ $(8 866LL  1 1!99!**!.!;!;!;;$1$G$G(44(44%--(33)==$1$G$G >(55 0*88(44&3&K&K&>Mr)NNNN)Trxryrzr{rCreateJobFromTemplateRequestrLaunchTemplateParametersrParametersValuerLaunchFlexTemplateRequestrrFlexTemplateRuntimeEnvironmentrzrrxTransformNameMappingValuerLaunchFlexTemplateParameterrrTransformNameMappingsValuerrryFlexrsGoalValueValuesEnumrTemplateMetadatar+SDKInfor]LanguageValueValuesEnumr^ ContainerSpecr{r'rPrQrRrSrTrUrWrXrYr9rS_IRWXUS_IRWXGS_IRWXOrS_IWUSRS_IRUSRS_IRGRPS_IROTHrS_IXGRPS_IXOTHrrr-rrrrrrrrr%rr1r7r5rr_rkrrrrrrrrrrs4$&CC.02KK%?%O%O"!2!4!N!N#33/1PP99"!::0./KK#:#J#J 88-!2!""#A#A,""#<#<')::  ( (("$,,DD,$&44.$5h$?!$5h$?!$5h$?!/@4/P,/@4/P,/@4/P,%6y%A" 1$ 7+= level start_time: If specified, return only messages with timestamps >= start_time. The default is the job creation time (i.e. beginning of messages). end_time: Return only messages with timestamps < end_time. The default is now (i.e. return up to the latest messages available). page_size: If specified, determines the maximum number of messages to return. If unspecified, the service may choose an appropriate default, or may return an arbitrarily large number of results. page_token: If supplied, this should be the value of next_page_token returned by an earlier call. This will cause the next page of results to be returned. Returns: (ListJobMessagesResponse) )r0r1r2rendTimeminimumImportancepageSize pageTokenN) r#r4r0DataflowProjectsLocationsJobsMessagesListRequestrr-Listr6r7rr8) r9r:r;minimum_importancerend_time page_size page_tokenr<r=s rrz Messages.ListVs>+z|J88I66,7 ,  " ' ' 00  ( (,  $ $U ++,s"AB 1BB )NNNNNNN) rxryrzr{rrr~rr-rrrrrrLsW3"44@@".,.,rrcVeZdZdZedZeddZeddZeddZy) SnapshotszCloud Dataflow snapshots api.c*tjSr)rprojects_locations_snapshotsrrrr-zSnapshots.GetServicerrNc|xs t}|xst}tj|||} tj j |S#tj$r}tj|d}~wwxYw)a Calls the Dataflow Snapshots.Delete method. Args: snapshot_id: The id of the snapshot to delete. project_id: The project that owns the snapshot. region_id: The regional endpoint of the snapshot. Returns: (DeleteSnapshotResponse)  snapshotIdr1r2N) r#r4r/DataflowProjectsLocationsSnapshotsDeleteRequestrr-Deleter6r7rr8 snapshot_idr:r;r<r=s rrzSnapshots.Deletes+z|J88I55j6J ,  ! ! # * *7 33  ( (,  $ $U ++,"AB,BBc|xs t}|xst}tj|||} tj j |S#tj$r}tj|d}~wwxYw)aCalls the Dataflow Snapshots.Get method. Args: snapshot_id: The id of the snapshot to get. project_id: The project that owns the snapshot. region_id: The regional endpoint of the snapshot. Returns: (GetSnapshotResponse) rN) r#r4r,DataflowProjectsLocationsSnapshotsGetRequestrr-r"r6r7rr8r s rr"z Snapshots.Gets+z|J88I!NNjOJG,  ! ! # ' ' 00  ( (,  $ $U ++,r c|xs t}|xst}tj|||} tj j |S#tj$r}tj|d}~wwxYw)a0Calls the Dataflow Snapshots.List method. Args: job_id: If specified, only snapshots associated with the job will be returned. project_id: The project that owns the snapshot. region_id: The regional endpoint of the snapshot. Returns: (ListSnapshotsResponse) )r0r1r2N) r#r4r-DataflowProjectsLocationsSnapshotsListRequestrr-rr6r7rr8)r9r:r;r<r=s rrzSnapshots.Lists+z|J88I!OOyJP@G,  ! ! # ( ( 11  ( (,  $ $U ++,r rv) rxryrzr{rr-rr"rrrrrrsS%<<,,,,,*,,rr)F)*r{ __future__rrrr=rcrrrapitools.base.pyrrr6!googlecloudsdk.api_lib.cloudbuildrgooglecloudsdk.api_lib.storager r googlecloudsdk.api_lib.utilr !googlecloudsdk.command_lib.buildsr googlecloudsdk.corer rgooglecloudsdk.core.utilrrrrr4rrr#r'r)rrrrrrrrrs;&' %>=67,29#** +II;\`,`,F$,$,NR#R#jg,g,T9,9,xI,I,r