C@dZddlmZddlmZddlmZddlZddlmZddlm Z ddlm Z dd lm Z dd l m Z d Zd Ze j e j"e j$j&e j$j(Gd de j*Zy)z?Implementation of gcloud dataflow flex_template build command. )absolute_import)division)unicode_literalsN)apis)actions) arg_parsers)base) propertiesc ~ |jd}|j}|j}|jdddtjdd|jd d |jd d tjdd|jddtjdd|jddtjdd|jddgdd|jddtj |jdddt jtjjj|jdd tjdd|jd!d"tjdd|jd#tjd$d%d&'|jd(tjd$d%d)'|jd*td+'|jd,t jtjjjd-.|jd/td0'|jd1d2 |jd3d4 |jd5d6 |jd7d8 |j}|jd9d: |jd;d< |jd=t jtjjjd>.|jd?d@tjdddAB|jdCdDtjtj dEF|jdGdHtj"tj dIF|jdJdKtjdLdMdN|jd}|jdOdPtjtj dQF|jdRdStjtj dTF|jdUdVdWX|jdYddZtj d[\|jd]d^d_X|jd`datjdbdcdN|jdddetj"tj dfdgyA)hzmRegisters flags for this command. Args: parser: argparse.ArgumentParser to register arguments with. T)requiredtemplate_file_gcs_pathTEMPLATE_FILE_GCS_PATHz]The Google Cloud Storage location of the flex template file.Overrides if file already exists.z^gs://.*zMust begin with 'gs://')metavarhelptypez--imagezLPath to the any image registry location of the prebuilt flex template image.)rz%--image-repository-username-secret-idaSecret Manager secret id for the username to authenticate to private registry. Should be in the format projects/{project}/secrets/{secret}/versions/{secret_version} or projects/{project}/secrets/{secret}. If the version is not provided latest version will be used.zD^projects\/[^\n\r\/]+\/secrets\/[^\n\r\/]+(\/versions\/[^\n\r\/]+)?$z~Must be in the format 'projects/{project}/secrets/{secret}' or'projects/{project}/secrets/{secret}/versions/{secret_version}'.)rrz%--image-repository-password-secret-idaSecret Manager secret id for the password to authenticate to private registry. Should be in the format projects/{project}/secrets/{secret}/versions/{secret_version} or projects/{project}/secrets/{secret}. If the version is not provided latest version will be used.z--image-repository-cert-patha)The full URL to self-signed certificate of private registry in Cloud Storage. For example, gs://mybucket/mycerts/selfsigned.crt. The certificate provided in Cloud Storage must be DER-encoded and may be supplied in binary or printable (Base64) encoding. If the certificate is provided in Base64 encoding, it must be bounded at the beginning by -----BEGIN CERTIFICATE-----, and must be bounded at the end by -----END CERTIFICATE-----. If this parameter is provided, the docker daemon in the template launcher will be instructed to trust that certificate. z--sdk-languagez&SDK language of the flex template job.)JAVAPYTHONGOYAML)rchoicesr z--metadata-filez;Local path to the metadata json file for the flex template.z --print-onlyzKPrints the container spec to stdout. Does not save in Google Cloud Storage.F)rdefaultactionz--staging-locationzbDefault Google Cloud Storage location to stage local files.(Must be a URL beginning with 'gs://'.)z--temp-locationzDefault Google Cloud Storage location to stage temporary files. If not set, defaults to the value for --staging-location.(Must be a URL beginning with 'gs://'.)z--service-account-emailz .*@.*\..*z"must provide a valid email addressz.Default service account to run the workers as.)rrz--cloud-build-service-accountaService account to run the Cloud Build in the format projects/{project}/serviceAccounts/{service_account}. Ensure that the account you are using to run 'gcloud dataflow flex-template build' has 'ServiceAccountUser' role on the specified Cloud Build service account you provide with the --cloud-build-service-account flag. The specified service account must have required permissions to build the image. If the specified service account is in a project that is different from the project where you are starting builds, see https://cloud.google.com/build/docs/securing-builds/configure-user-specified-service-accounts#cross-project_set_up to grant the necessary access.z --max-workersz)Default maximum number of workers to run.z--disable-public-ipsz8Cloud Dataflow workers must not use public IP addresses.)rrz --num-workersz,Initial number of workers to use by default.z--worker-machine-typezIDefault type of machine to use for workers. Defaults to server-specified.z --subnetworkzODefault Compute Engine subnetwork for launching instances to run your pipeline.z --networkzLDefault Compute Engine network for launching instances to run your pipeline.z--dataflow-kms-keyz3Default Cloud KMS key to protect the job resources.z--worker-regionz%Default region to run the workers in.z --worker-zonez#Default zone to run the workers in.z--enable-streaming-enginez9Enable Streaming Engine for the streaming job by default.z --gcs-log-dirzYGoogle Cloud Storage directory to save build logs.(Must be a URL beginning with 'gs://'.)N)rrrz--additional-experimentsADDITIONAL_EXPERIMENTSz'Default experiments to pass to the job.)rrrrz--additional-user-labelsADDITIONAL_USER_LABELSz]Default user labels to pass to the job. Example: --additional-user-labels='{"key1":"value1"}'z--image-gcr-pathzpThe Google Container Registry or Google Artifact Registry location to store the flex template image to be built.z0^(.*\.){0,1}gcr.io/.*|^(.){2,}-docker.pkg.dev/.*aMust begin with '[multi-region.]gcr.io/' or '[region.]-docker.pkg.dev/'. Please check https://cloud.google.com/container-registry/docs/overview for available multi-regions in GCR or https://cloud.google.com/artifact-registry/docs/repo-organize#locations for available location in GAR)rrr z--jarJARa Local path to your dataflow pipeline jar file and all their dependent jar files required for the flex template classpath. You can pass them as a comma separated list or repeat individually with --jar flag. Ex: --jar="code.jar,dep.jar" or --jar code.jar, --jar dep.jar.z --py-pathPY_PATHa<Local path to your dataflow pipeline python files and all their dependent files required for the flex template classpath. You can pass them as a comma separated list or repeat individually with --py-path flag. Ex: --py-path="path/pipleline/,path/dependency/" or --py-path path/pipleline/, --py-path path/dependency/.z--go-binary-pathGO_BINARY_PATHzLocal path to your compiled dataflow pipeline Go binary. The binary should be compiled to run on the target worker architecture (usually linux-amd64). See https://beam.apache.org/documentation/sdks/go-cross-compilation/ for more information.)rrz--yaml-pipeline-pathYAML_PIPELINE_PATHz&Local path to your YAML pipeline file.)r rrrz --yaml-image YAML_IMAGEzLPath to the any image registry location of the prebuilt yaml template image.z--flex-template-base-imageaFlex template base image to be used while building the container image. Allowed choices are allowed labels (JAVA11, JAVA17, JAVA21, PYTHON3, GO), supported distroless images (JAVA11_DISTROLESS, JAVA17_DISTROLESS, JAVA21_DISTROLESS, GO_DISTROLESS), or full gcr.io path of the specific version of the base image. For labels, we use the latest base image version to build the container. You can also provide a specific version from this link https://gcr.io/dataflow-templates-base/zF^(JAVA11|JAVA17|JAVA21|GO)(_DISTROLESS)?$|^JAVA8$|^PYTHON3$|^gcr.io/.*zMust be JAVA11, JAVA17, JAVA21, PYTHON3, GO, (or with `_DISTROLESS` suffix for supported distroless variants), or begin with 'gcr.io/'z--envENVaXEnvironment variables to create for the Dockerfile. You can pass them as a comma separated list or repeat individually with --env flag. Ex: --env="A=B,C=D" or --env A=B, --env C=D.When you reference files/dir in env variables, please specify relative path to the paths passed via --py-path.Ex: if you pass. --py-path="path/pipleline/" then set FLEX_TEMPLATE_PYTHON_PY_FILE="pipeline/pipeline.py" You can find the list of supported environment variables in this link. https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#setting_required_dockerfile_environment_variables.)rrrrr )add_mutually_exclusive_groupadd_argument_group add_argumentrRegexpValidator FileContentsrStoreBooleanPropertyr VALUESdataflow print_onlyintdisable_public_ipsenable_streaming_engineArgList UpdateActionArgDict)parser image_argsimage_building_args yaml_args region_group pipeline_argss +lib/surface/dataflow/flex_template/build.py _CommonArgsr7s% 22D2A*"557++-)& 0  & &{'B D E    - 4  & & Q O P Q - 4  & & Q O P Q $ 5  & &{'B D E  3.   H  # # %'   $  ) )    $ $ / /1 2  6  & &{'B D E  6  & &{'B D E   & &|'K M ; =  %  & & <  , (   6    ) )    $ $ 7 79 E G    9;            @B446, 24 02 !  ) )    $ $ < <> F H   6  & &{'B D   &     % %0 3  &     % % ;  "" E  & & = 5 7# &BBC-      % % -  /     % % E  G $& "  # # % 3    """ 5  & & S   )#.""      % % =##c>tj|j|j|j|j |j |j|j|jtjjjj|j|j |j"tjjj$j|j&|j(}|j*dk(r|j,s t/dt1j2|j4}|j,|d<t1j6|d|_tj8j;|}n |j<}|s|j>}tj8jA|j>|jB|jD|jF|jH|jJ|j*|jL|jN tj8jQ|jR||j4|j*|jT||jV|jX|jZ S)zRuns the command. Args: args: The arguments that were provided to this command invocation. Returns: A Job message. ) max_workers num_workersnetwork subnetworkworker_machine_type kms_key_namestaging_location temp_locationr+service_account_email worker_region worker_zoner,additional_experimentsadditional_user_labelsrzyaml_pipeline_path is required.yamlDefinition)indent).rTemplateArgumentsr:r;r<r=r>dataflow_kms_keyr@rAr r'r(r+GetBoolrBrCrDr,rErF sdk_languageyaml_pipeline_path ValueErrorjsonloads metadata_filedumps TemplatesGetYamlTemplateImageimageimage_gcr_pathBuildAndStoreFlexTemplateImageflex_template_base_imagejarpy_pathgo_binary_pathenv gcs_log_dircloud_build_service_accountBuildAndStoreFlexTemplateFiler r)#image_repository_username_secret_id#image_repository_password_secret_idimage_repository_cert_path)args template_argsmetadata image_paths r6 _CommonRunrhSs((""""ll22((,,&&#**33FFNN 66&&""(//88PPwy!88!88#:-& &  " " 8 99zz$,,-H!%!8!8H HQ7D44T:JJ $$JNN11  %%       ((   5 5 !!:t/A/A - .. ..0O0O  QQr8c0eZdZdZdddZedZdZy)Buildz:Builds a flex template file from the specified parameters.az To build and store a flex template JSON file, run: $ {command} gs://template-file-gcs-path --image=gcr://image-path --metadata-file=/local/path/to/metadata.json --sdk-language=JAVA If using prebuilt template image from private registry, run: $ {command} gs://template-file-gcs-path --image=private.registry.com:3000/image-path --image-repository-username-secret-id="projects/test-project/secrets/username-secret" --image-repository-password-secret-id="projects/test-project/secrets/password-secret/versions/latest" --metadata-file=metadata.json --sdk-language=JAVA To build the template image and flex template JSON file, run: $ {command} gs://template-file-gcs-path --image-gcr-path=gcr://path-to-store-image --jar=path/to/pipeline.jar --jar=path/to/dependency.jar --env=FLEX_TEMPLATE_JAVA_MAIN_CLASS=classpath --flex-template-base-image=JAVA11 --metadata-file=/local/path/to/metadata.json --sdk-language=JAVA ) DESCRIPTIONEXAMPLESct|yN)r7)r0s r6Argsz Build.Argss r8ct|Srn)rh)selfrds r6Runz Build.Runs d r8N)__name__ __module__ __qualname____doc__ detailed_help staticmethodrorrr8r6rjrjs3C G  -<r8rj)rv __future__rrrrPgooglecloudsdk.api_lib.dataflowrgooglecloudsdk.callioperrr googlecloudsdk.corer r7rhDefaultUniverseOnly ReleaseTracks ReleaseTrackGABETACommandrjryr8r6rs&' 0+/(*rj 8QvD%%(($*;*;*@*@A&DLL&B&r8