)kdZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddlmZddlm ZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlm Z ddl!m"Z"ddl#m$Z$ddl#m%Z%ddl&Z&dZ'dZ(e jRdZ*dZ+dZ,dZ-dZ.iZ/Gddej`ddZ1ide1dd d!e1dd d"e1dd# d$e1d%d&e1d%d& e1d'd& d()d*e1dd d+e1d,d# d-e1d.d d/e1dd# d0e1d1d# d2e1d3d d4e1dd d5e1dd& d6e1dd d7e1d3d d8e1dd d9e1dd d:e1d,d# e1dd e1dd# e1dd e1d,d# e1d3d e1dd e1dd e1d,d# e1dd e1dd e1dd e1d;d e1d,d# e1dd# e1dd# d<Z2e2jge/d=Z4d>Z5d?Z6d@Z7ej`dAdBZ8GdCdDejrZ9GdEdFe9Z:GdGdHe9Z;GdIdJe9Z<GdKdLe9Z=GdMdNe9Z>GdOdPe9Z?dQZ@ejdadRZBdSZCdbdTZDdUZEdcdVZFdWZGdXZHdYZIdZZJd[ZKd\ZLd]ZMd^ZNejjfd_ZQd`ZRy)dz'Common utilities for Composer commands.)absolute_import)division)unicode_literalsN)util) api_adapter) storage_api) storage_util)base)parsers)config) exceptions)execution_utils)log) console_io)resource_printer)encoding)filesv1 KUBECONFIGz#^[a-z](?:[-0-9a-z]{0,62}[0-9a-z])?$gcloudkubectlz!Unable to find [gcloud] on path! zAccessing a Cloud Composer environment requires the kubernetes commandline client [kubectl]. To install, run $ gcloud components install kubectl c(eZdZdZ dfd ZxZS)SupportedAirflowVersionc0tt| ||||SN)superr__new__)cls from_version to_versionallowed_nested_subcommands __class__s /lib/googlecloudsdk/command_lib/composer/util.pyrzSupportedAirflowVersion.__new__Rs$ (# 6 \:'A CC)N3.0.0N)__name__ __module__ __qualname__ __slots__r __classcell__r#s@r$rrLs)18)-CCr%rz2from_version to_version allowed_nested_subcommandsbackfillz2.0.0)r r!clear connectionsz3.1.0dbz2.3.0r&z2.6.3)checktrim)r r!r" dag_statedagsz1.10.14 delete_dagz1.10.1kerberos kubernetesz2.1.4 list_dag_runsz1.10.2 list_dagszlist-import-errors list_tasksnext_executionpausepoolpoolsz1.10.15)renderrolesrunz sync-perm sync_permtask_failed_deps task_statetaskstest trigger_dagunpause upgrade_checkusers variablesversiondefaultz --namespacez-nactive GkePodStatuszname phase isReadyceZdZdZy)Errorz-Class for errors raised by Composer commands.Nr'r(r)__doc__rr%r$rQrQs5r%rQceZdZdZy) KubectlErrorz5Class for errors raised when shelling out to kubectl.NrRrr%r$rUrU=r%rUceZdZdZy) GsutilErrorz4Class for errors raised when shelling out to gsutil.NrRrr%r$rXrXsg(([Ak"";/g   dMD$:$:Mt~~Nm x55HHJ''-vk.C.C'DF F$$    3% rzz+B-/+"!*rzz+B-/+"!s6)G&A G5DF,9)G# G&,+GGG#G&c \|jjjrS|jjj|jjjjddzdS|jj|jjjddzd}t j t}|jtjjDcgc]J}|j|k(r9|j|jjddzdxs |jL}}|s tdt|dk(r|dS|t!j"|Dcgc]}dj%|c}ddScc}wcc}w) a|Finds the location ID of the GKE cluster running the provided environment. Args: env_object: Environment, the environment, likely returned by an API call, whose cluster location to extract Raises: Error: if Kubernetes cluster is not found. Returns: str, the location ID (a short name like us-central1-b) of the GKE cluster running the environment /Nz$Kubernetes Engine cluster not found.rz[{}]zLCluster found in more than one location. Please select the desired location:)rMmessage)r nodeConfiglocationrfind gkeClusterr~rr ListClustersr GetProjectclustersnamezonerQlenr PromptChoicer]) env_object gke_clusterrc cluster_zoneszs r$ExtractGkeClusterLocationIdrs!!**    ' ' 0 01B1B1M1M2:(55:2J2K LL!!,,Z->->-I-I.3eCj1.=.>?+  ) )/ :' ##G$6$6$89BBB! ; jj!!#&*+,66B  6 77 =Q   z..!./Av}}Q/ 0s ;AF$F) ctj}gd} t||jtj j||jjdDcgc]}|rt|jd}}|jd|Dcgc]+}|jjd k(r |j-}}|s td |d S tfd |DS#t $r}td|zd}~wwxYwcc}wcc}w#t $r td wxYw)aReturns the name of a running pod in a GKE cluster. Retrieves pods in the GKE cluster pointed to by the current kubeconfig context. To target a specific cluster, this command should be called within the context of a TemporaryKubeconfig context manager. If pod_substr is not None, the name of an arbitrary running pod whose name contains pod_substr is returned; if no pod's name contains pod_substr, an Error is raised. If pod_substr is None, an arbitrary running pod is returned. Pods with 'Ready: true' condition state are preferred. If there are no such pods, any running pod will be returned. Args: pod_substr: string, a filter to apply to pods. The returned pod name must contain pod_substr (if it is not None). kubectl_namespace: string or None, namespace to query for gke pods Raises: Error: if GKE pods cannot be retrieved or desired pod is not found. )getpods--outputzyjsonpath={range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.conditions[?(.type=="Ready")].status}{"\n"})out_funcerr_func namespacezError retrieving GKE pods: %sN  c<|jjdk7S)Ntrue)isReadylower)xs r$zGetGkePod..os!))//"3v"=r%)keyrunningzZNo running GKE pods found. If the environment was recently started, please wait and retry.rc3,K|] }|vs| ywrr).0pod pod_substrs r$ zGetGkePod..sA|zS/@|s zZDesired GKE pod not found. If the environment was recently started, please wait and retry.)rirjRunKubectlCommandwritererrrUrQrlsplitrOsortphaserrnext StopIteration)rkubectl_namespacepod_outargse pod_status cluster_pods running_podss` r$ GetGkePodr?sk. KKM' $ 5 # % ((*0066* J$$T*+6=>%$*      !Y .oo$  ? @@ ?@ A|A AA9 5 /!3 445 @ ? @@@s/1D- D &0D%-D* D DD*D?c0tj|duS)z>Returns True if the provided name is a valid environment name.N)ENVIRONMENT_NAME_PATTERNmatch)rs r$IsValidEnvironmentNamers ! ' ' -T 99r%ctjttjj }|tjt}|t tt|tj|g|} tj|d|fdd}|r tdy#tjtjf$r#}ttj |d}~wwxYw)aShells out a command to kubectl. This command should be called within the context of a TemporaryKubeconfig context manager in order for kubectl to be configured to access the correct cluster. Args: args: list of strings, command line arguments to pass to the kubectl command. Should omit the kubectl command itself. For example, to execute 'kubectl get pods', provide ['get', 'pods']. out_func: str->None, a function to call with the stdout of the kubectl command err_func: str->None, a function to call with the stderr of the kubectl command namespace: str or None, the kubectl namespace to apply to the command Raises: Error: if kubectl could not be called KubectlError: if the invocation of kubectl was unsuccessful NTct|Sr)HandleKubectlErrorStream)rrs r$rz#RunKubectlCommand..s 5hDr%)no_exitrruniversal_newlinesz&kubectl returned non-zero status code.)rFindExecutableOnPath_KUBECTL_COMPONENT_NAMEr Paths sdk_bin_pathrQMISSING_KUBECTL_MSGAddKubectlNamespacerArgsForExecutableToolExecPermissionErrorInvalidCommandErrorrUsix text_type)rrrr kubectl_path exec_argsretvalrs ` r$rrs.++,C,2LLN,G,GI,--.EFL # $$!66|KdKM) ) ! !D !F ? @@   ) )  - - /) s}}Q' (()sB00#C6C11C6cz|xstjj}d|vr|||dy||y)NzUnable to connect to the serverz> Please, check if you have connectivity to GKE control plane. )rstatusrk)rrerr_handler_funcs r$rrs=1!1!1&#-SJLSr%c&|jddS)zBConverts an image version string to a kubernetes namespace string..-)replace) image_versions r$$ConvertImageVersionToNamespacePrefixrs   sC ((r%ct|}gd}tj}t||jt j jt|jjd}|D]^}|jr|jdnd}|s)|djtk(sD|dj|sY|dcStS)alChecks environment for valid namespace options. First checks for the existence of a kubectl namespace based on the env image version. If namespace does not exist, then return the 'default' namespace. Args: env_image_version: str, the environment image version string. Returns: The namespace string to apply to any `environments run` commands. )rrz--all-namespacesz%--sort-by=.metadata.creationTimestamprzEjsonpath={range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}z--ignore-not-found=truerrNrr)rrirjrrrrreversedrlrstriprNAMESPACE_STATUS_ACTIVE startswithDEFAULT_NAMESPACE)env_image_versionimage_version_ns_prefixr ns_output namespacesns_entryns_partss r$FetchKubectlNamespacersA $kkm)D)//377==9 **,22489*h'/~~'7x~~d#THXa[&&(,CC 67 a[  r%c||Stthjt|r0d}|r t|dvrd}|tfD]}|j |||S)arAdds namespace arguments to the provided list of kubectl args. If a namespace arg is not already present, insert `--namespace ` after the `kubectl` command and before all other arg elements. Resulting in this general format: ['kubectl', '--namespace', 'namespace_foo', ... ... ] Args: namespace: name of the namespace scope kubectl_args: list of kubectl command arguments. Expects that the first element will be the `kubectl` command, followed by all additional arguments. Returns: list of kubectl args with the additional namespace args (if necessary). rr)NAMESPACE_ARG_NAMENAMESPACE_ARG_ALIAS isdisjointsetrinsert)r kubectl_argsidxnew_args r$rrsn$ -.99#l:KL C/<?B c12#w'3 r%c |jd}|rEtj}tjj |}|j |}ntj|}i}|5}|D]W}|j}|r|jdr't|\}} ||vrtdj|| ||<Y|cdddS#1swYyxYw#tjtjtjf$r0tjtdj|YywxYw)aParses the given requirements file into a requirements dictionary. If the file path is GCS file path, use GCS file parser to parse requirements file. Otherwise, use local file parser. Args: requirements_file_path: Filepath to the requirements file. Returns: {string: string}, dict mapping from PyPI package name to extras and version specifier, if provided. Raises: Error: if requirements file cannot be read. zgs://#z+Duplicate package in requirements file: {0}Nz$Unable to read requirements file {0})rr StorageClientr ObjectReferenceFromUrl ReadObjectr FileReaderrSplitRequirementSpecifierrQr]core_exceptionsreraise) requirements_file_pathis_gcs_file_pathstorage_client object_ref file_content requirementsrequirements_filerequirement_specifierpackagerLs r$ParseRequirementsFilers6 &-88A"002n//778NOj#..z:l%%&<=lL *#4  5 ; ; =$(=(H(H(M 45JK l ";BB7KM M ' W$5  ++{((,*<*< =& 4;; "$ %&&s2A1C'3AC C'C$ C'$C''AE E c|j}d}tj||}d}|rB|d|jj}||jdj}|st dj |||fS)aISplits the package name from the other components of a requirement spec. Only supports PEP 508 `name_req` requirement specifiers. Does not support requirement specifiers containing environment markers. Args: requirement_specifier: str, a PEP 508 requirement specifier that does not contain an environment marker. Returns: (string, string), a 2-tuple of the extracted package name and the tail of the requirement specifier which could contain extras and/or a version specifier. Raises: Error: No package name was found in the requirement spec. z(\[|\(|==|>=|!=|<=|<|>|~=|===)Nz5Missing package name in requirement specifier: \'{}\')rresearchstartrQr])rrtail_start_regex tail_matchtails r$r r @s$ " ' ' )'6yy)+@A* $#$7Z%5%5%78>>@G !1!1!3!4 5 ; ; =D  HOO  $r%c td|xsgD}tjdtt j |xsiD}|r9t j |Dcgc]\}}|||}}}|||fSg} t} |D]4}| j dj||| j|6g}t j |D]D\}}|j ||||| vs$| j dj||F| jdj| ||fScc}}w)aBuilds the field mask and patch environment for an environment update. Follows the environments update semantic which applies operations in an effective order of clear -> remove -> set. Leading and trailing whitespace is stripped from elements in remove_keys and the keys of set_entries. Args: clear: bool, If true, the patch removes existing keys. remove_keys: iterable(string), Iterable of keys to remove. set_entries: {string: string}, Dict containing entries to set. field_mask_prefix: string, The prefix defining the path to the base of the proto map to be patched. entry_cls: AdditionalProperty, The AdditionalProperty class for the type of entry being updated. env_builder: [AdditionalProperty] -> Environment, A function which produces a patch Environment with the given list of entry_cls properties. Returns: (string, Environment), a 2-tuple of the field mask defined by the arguments and a patch environment produced by env_builder. c3<K|]}|jywrrrks r$rz%BuildPartialUpdate..x9'8!AGGI'8c3FK|]\}}|j|fywrr$rr&vs r$rz%BuildPartialUpdate..{$(K!IAqwwy!n!I!rvaluez{}.{},) r collections OrderedDictsortedr iteritemsappendr]addrry) r. remove_keys set_entriesfield_mask_prefix entry_cls env_builderrr/entriesfield_mask_entries seen_keyss r$BuildPartialUpdater?_sY29{'8b'899+''(K!' k6GR(H!I(KK+ -- 44JC c'4  k'2 22e) cgnn->DE MM# 'MM+.jc5 NN9E23 )/@# FG/  $ %{7'; ;;'s-Ec tjd|D}|rtj}td|xsgD}|D] }||vs||= tjdtt j |xsiD}|j ||t j |Dcgc]\}}|||c}}Scc}}w)a:Builds the patch environment for an environment update. To be used when BuildPartialUpdate cannot be used due to lack of support for field masks containing map keys. Follows the environments update semantic which applies operations in an effective order of clear -> remove -> set. Leading and trailing whitespace is stripped from elements in remove_keys and the keys of set_entries. Args: clear: bool, If true, the patch removes existing keys. remove_keys: iterable(string), Iterable of keys to remove. set_entries: {string: string}, Dict containing entries to set. initial_entries: [AdditionalProperty], list of AdditionalProperty class with key and value fields, representing starting dict to update from. entry_cls: AdditionalProperty, The AdditionalProperty class for the type of entry being updated. env_builder: [AdditionalProperty] -> Environment, A function which produces a patch Environment with the given list of entry_cls properties. Returns: Environment, a patch environment produced by env_builder. c3LK|]}|j|jfywrr.)rentrys r$rz%BuildFullMapUpdate..s!)=,;5uyy%++Os"$c3<K|]}|jywrr$r%s r$rz%BuildFullMapUpdate..r'r(c3FK|]\}}|j|fywrr$r*s r$rz%BuildFullMapUpdate..r,r-r.)r1r2rr3rr4update) r.r7r8initial_entriesr:r; entries_dictrr/s r$BuildFullMapUpdaterHs8(()=,;)==, **,L9{'8b'899+ c l s  ''(K!' k6GR(H!I(KK+k"  l33*#uCu%3 s?C ctj|jjj}|j |k(S)a=Returns whether an environment currently is in the RUNNING state. Args: environment: Environment, an object returned by an API call representing the environment to check. release_track: base.ReleaseTrack, the release track of command. Will dictate which Composer client library will be used. ) release_track)api_utilGetMessagesModule EnvironmentStateValueValuesEnumRUNNINGstate) environmentrJ running_states r$IsInRunningStaterSs<  %''2{3G3G   m ++r%c|y|D]} tj|y#t$r}tdj |d}~wwxYw)zValidates given master authorized networks. Args: networks: Iterable(string) or None. List of networks in CIDR notation. Nz%Invalid master authorized network: {}) ipaddress IPv4Network Exceptionrfr])networksnetworkrs r$ ValidateMasterAuthorizedNetworksrZsW  g=G$ = ! 1 8 8 ; ===s" A AA r)NN)NNN)SrS __future__rrrr1 contextlibrirUrwrgooglecloudsdk.api_lib.composerrrK googlecloudsdk.api_lib.containerrr~rtgooglecloudsdk.api_lib.storagerr googlecloudsdk.callioper #googlecloudsdk.command_lib.composerr googlecloudsdk.corer r rrrgooglecloudsdk.core.consolergooglecloudsdk.core.resourcergooglecloudsdk.core.utilrrrrr|compiler_GCLOUD_COMPONENT_NAMErMISSING_GCLOUD_MSGrSUBCOMMAND_DEPRECATION namedtuplerSUBCOMMAND_ALLOWLISTrErrrrrOrQrUrXrZrbrdrfrpcontextmanagerrrrrrrrrrrr r?rH ReleaseTrackGArSrZrr%r$ros.&'  <K=67(7&=/#29-* &%2::&KL!# C4k448: CKTgFK TgF K TgF K  ,%,B,%,B ( K&TgF)K* Y7K-K.X'J1K2TgF5K6WI9K:X'J=K>TgFAKBTgFEKFTgFIKJX'JMKN TgFQKR TgFUKV Y7KYK\ TgFWITgFY7KX'JTgFTgFY7KTgFTgFTgFY7KY7KTgFTgFUK^23""%{%%n6JK 6O ! !6>5>=%=&U&>U>>U>PEP   ////d(VC@L: /Ad) "J F*&Z>2