UJdZddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd l m Z dd lmZej ej"ej$j&ej$j(Gd d ej*Zd ZdZdZy)z3Implementation of gcloud dataflow yaml run command.)apis) storage_api) storage_util) arg_parsers)base) dataflow_util) properties)yaml)filesc0eZdZdZdddZedZdZy)Runz#Runs a job from the specified path.zERuns a job from the specified YAML description or Cloud Storage path.z To run a job from YAML, run: $ {command} my-job --yaml-pipeline-file=gs://yaml-path --region=europe-west1 ) DESCRIPTIONEXAMPLESc|jddd|jd}|jdd |jd d |jd d dtjz|jddt j tj d|jddd|jddt jdd|jdd |jdd y) zrRegister flags for this command. Args: parser: argparse.ArgumentParser to register arguments with. job_nameJOB_NAMEz!Unique name to assign to the job.)metavarhelpTrequiredz--yaml-pipeline-fileziPath of a file defining the YAML pipeline to run. (Must be a local file or a URL beginning with 'gs://'.))rz--yaml-pipelinez.Inline definition of the YAML pipeline to run.z--region REGION_IDz*Region ID of the job's regional endpoint. z--pipeline-optionszOPTIONS=VALUE;OPTION=VALUEz$Pipeline options to pass to the job.)rtypeactionrz--jinja-variables JSON_OBJECTz1Jinja2 variables to be used in reifying the yaml.z--template-file-gcs-locationzbGoogle Cloud Storage location of the YAML template to run. (Must be a URL beginning with 'gs://'.)z^gs://.*zMust begin with 'gs://')rrz --networkztCompute Engine network for launching worker instances to run the pipeline. If not set, the default network is used.z --subnetworkzzCompute Engine subnetwork for launching worker instances to run the pipeline. If not set, the default subnetwork is used.N) add_argumentadd_mutually_exclusive_grouprDEFAULT_REGION_MESSAGErArgDict UpdateActionRegexpValidator)parsergroups lib/surface/dataflow/yaml/run.pyArgszRun.Args-sU J-P  / / / >E  F  P  822 3  ,  "'' 3   @  &8  ( ()DF   F  Mc t|jxsi}|jr-t|j}||j|d<n||d<n|j|d<|j r|j |d<d|vrd|vrt |dt|jd|}|jxs)tjjj|}tjtj j"j$j'd||j(|||j*|j,}tjj/|S)zRuns the command. Args: args: The arguments that were provided to this command invocation. Returns: A Job message. yaml_pipeline_file yaml_pipelinejinja_variableszjinja-variablesTr) project_id region_idr gcs_location parametersnetwork subnetwork)dictpipeline_optionsr'_try_get_yaml_contentsr(r)_validate_yaml _get_region_from_yaml_or_defaultgettemplate_file_gcs_locationr TemplatesYAML_TEMPLATE_GCS_LOCATIONformatTemplateArgumentsr VALUEScoreprojectGetrr.r/CreateJobFromFlexTemplate)selfargsr- yaml_contentsr+r, argumentss r#r zRun.RuntsNd++1r2J ,T-D-DEm  +/+B+B '(&3 ?#%)$6$6j! &*&:&:j"#*$):*)LZ010'I '' G >> 4 4 ; ;I F &&$$))1155t5D! ??I >> 3 3I >>r%N)__name__ __module__ __qualname____doc__ detailed_help staticmethodr$r r%r#r r s4,  -DDL.?r%r cj tj|}y#t$r}td|d}~wwxYw)Nz#yaml_pipeline must be a valid yaml.)r load Exception ValueError)r(_exns r#r3r3s6E - A E : ;DEs 2 -2c|j}d} tj|}|stj|Sd|vr;d|dvr4|dd}d|vsd|vr$tj dj ||r$|r ||k7rtdj |||Stj|S#tj $r&}|stdj |Yd}~od}~wwxYw) zEGets the region from yaml pipeline or args, or falls back to default.Noptionsregion{}zlyaml pipeline contains unparsable region: {0}. Found curly braces in region. Falling back to default region.zyFailed to get region from yaml pipeline: {0}. If using jinja variables, parsing may fail. Falling back to default region.zeRegion specified in yaml pipeline options ({0}) does not match region specified in command line ({1})) rSr rLr GetRegionYAMLParseErrorr9printrN)r(rArSoptions_region pipeline_datarPs r#r4r4s ;;&.IIm,M   $ $T **M!h- 2J&J$Y/9n  #"7!! 99?9O   &N*  44:Ff5      &&%      F3Ks+B6?B66C/ C**C/ct|jdstj|Stj}t j j|} |j|jjdS#t$r%}tdj||Yd}~yd}~wwxYw)z@Reads yaml contents from the specified file if it is accessable.zgs://zutf-8zUnable to read file {0} due to incorrect file path or insufficient read permissions. Will not be able to validate the yaml pipeline or determine the region from the yaml pipeline options. Error: {1}N) startswithr ReadFileContentsr StorageClientrObjectReferenceFromUrl ReadObjectreaddecoderMrXr9)r'storage_clientobj_refes r#r2r2s  & &w /  ! !"4 55,,..  ( ( 0 01C D'  $ $W - 2 2 4 ; ;G DD    &v&8!<  s-B B7B22B7N)rGgooglecloudsdk.api_lib.dataflowrgooglecloudsdk.api_lib.storagerrgooglecloudsdk.callioperr#googlecloudsdk.command_lib.dataflowrgooglecloudsdk.corer r googlecloudsdk.core.utilr DefaultUniverseOnly ReleaseTracks ReleaseTrackGABETACommandr r3r4r2rJr%r#rss:067/(=*$*D%%(($*;*;*@*@AD?$,,D?BD?NE!'Hr%