KdZddlmZddlmZddlmZddlZddlmZddlmZdZ d Z Gd d Z Gd d Z y)z5Utilities for building the dataproc clusters gke CLI.)absolute_import)division)unicode_literalsN) arg_parsers) exceptionsc|jdtjddgttttjt ttttt t d dgdd y) N--poolsnameroles) r r machineType preemptible localSsdCount acceleratorsminCpuPlatformbootDiskKmsKey locationsminmax required_keysspecappendKEY=VALUE[;VALUE]a Each `--pools` flag represents a GKE node pool associated with the virtual cluster. It is comprised of a CSV in the form `KEY=VALUE[;VALUE]`, where certain keys may have multiple values. The following KEYs must be specified: ----------------------------------------------------------------------------------------------------------- KEY Type Example Description ------ ---------------- ------------------------ ---------------------------------------------------------- name string `my-node-pool` Name of the node pool. roles repeated string `default;spark-driver` Roles that this node pool should perform. Valid values are `default`, `controller`, `spark-driver`, `spark-executor`. ----------------------------------------------------------------------------------------------------------- The following KEYs may be specified: ---------------------------------------------------------------------------------------------------------------------------------------------------------------- KEY Type Example Description --------------- ---------------- --------------------------------------------- --------------------------------------------------------------------------------- machineType string `n1-standard-8` Compute Engine machine type to use. preemptible boolean `false` If true, then this node pool uses preemptible VMs. This cannot be true on the node pool with the `controllers` role (or `default` role if `controllers` role is not specified). localSsdCount int `2` The number of local SSDs to attach to each node. accelerator repeated string `nvidia-tesla-a100=1` Accelerators to attach to each node. In the format NAME=COUNT. minCpuPlatform string `Intel Skylake` Minimum CPU platform for each node. bootDiskKmsKey string `projects/project-id/locations/us-central1 The Customer Managed Encryption Key (CMEK) used to encrypt /keyRings/keyRing-name/cryptoKeys/key-name` the boot disk attached to each node in the node pool. locations repeated string `us-west1-a;us-west1-c` Zones within the location of the GKE cluster. All `--pools` flags for a Dataproc cluster must have identical locations. min int `0` Minimum number of nodes per zone that this node pool can scale down to. max int `10` Maximum number of nodes per zone that this node pool can scale up to. ---------------------------------------------------------------------------------------------------------------------------------------------------------------- typeactiondefaultmetavarhelp add_argumentrArgDictstr ArgBooleanintparsers 7lib/googlecloudsdk/command_lib/dataproc/gke_clusters.py AddPoolsArgr)ss     (335"! # #   &!+ 1C c|jdtjddgttttjt t ttttt t d dgdd y) Nr r r ) r r r r rlocalNvmeSsdCountrrrrrrrrra Each `--pools` flag represents a GKE node pool associated with the virtual cluster. It is a comma-separated list in the form `KEY=VALUE[;VALUE]`, where certain keys may have multiple values. The following KEYs must be specified: ----------------------------------------------------------------------------------------------------------- KEY Type Example Description ------ ---------------- ------------------------ ---------------------------------------------------------- name string `my-node-pool` Name of the node pool. roles repeated string `default;spark-driver` Roles that each node pool will perform. [One Pool must have DEFAULT role] Valid values are `default`, `controller`, `spark-driver`, `spark-executor`. ----------------------------------------------------------------------------------------------------------- The following KEYs may be specified: ---------------------------------------------------------------------------------------------------------------------------------------------------------------- KEY Type Example Description --------------- ---------------- --------------------------------------------- --------------------------------------------------------------------------------- machineType string `n1-standard-8` Compute Engine machine type to use. preemptible boolean `false` If true, then this node pool uses preemptible VMs. This Must be `false` for a node pool with the CONTROLLER role or for a node pool with the DEFAULT role in no node pool has the CONTROLLER role. localSsdCount int `2` The number of local SSDs to attach to each node. localNvmeSsdCount int `2` The number of local NVMe SSDs to attach to each node. accelerator repeated string `nvidia-tesla-a100=1` Accelerators to attach to each node, in NODE=COUNT format. minCpuPlatform string `Intel Skylake` Minimum CPU platform for each node. bootDiskKmsKey string `projects/project-id/locations/us-central1 The Customer Managed Encryption Key (CMEK) used to encrypt /keyRings/keyRing-name/cryptoKeys/key-name` the boot disk attached to each node in the node pool. locations repeated string `us-west1-a;us-west1-c` Zones within the location of the GKE cluster. All `--pools` flags for a Dataproc cluster must have identical locations. min int `0` Minimum number of nodes per zone that this node pool can scale down to. max int `10` Maximum number of nodes per zone that this node pool can scale up to. ---------------------------------------------------------------------------------------------------------------------------------------------------------------- rr r&s r(AddPoolsAlphaArgr-bsv     (335"#&! # #   (!. 3G r*cbeZdZdZeddZedZedZedZedZ y) GkeNodePoolTargetsParserzGParses all the --pools flags into a list of GkeNodePoolTarget messages.c |Dcgc]}tj||||}}tj|tj ||tj |tj ||Scc}w)aParses all the --pools flags into a list of GkeNodePoolTarget messages. Args: dataproc: The Dataproc API version to use for GkeNodePoolTarget messages. gke_cluster: The GKE cluster's relative name, for example, 'projects/p1/locations/l1/clusters/c1'. arg_pools: The list of dict[str, any] generated from all --pools flags. support_shuffle_service: support shuffle service. Returns: A list of GkeNodePoolTargets message, one for each entry in the arg_pools list. )_GkeNodePoolTargetParserParser/_ValidateUniqueNames_ValidateRoles_ValidatePoolsHaveSameLocation_ValidateBootDiskKmsKeyPattern)dataproc gke_cluster arg_poolssupport_shuffle_servicearg_poolpoolss r(r2zGkeNodePoolTargetsParser.Parses$" "H !&&xh'> @!   11%8++He<;;EB;;EB L sA=ct}|D]<}|j}||vrtjdd|z|j |>y)z'Validates that pools have unique names.r z#Pool name "%s" used more than once.N)setnodePoolrInvalidArgumentExceptionadd)r< used_namespoolr s r(r3z-GkeNodePoolTargetsParser._ValidateUniqueNamessRJ ]]d  11 r rr@rAmessagesGkeNodePoolTargetRolesValueListEntryValuesEnum)r7r< seen_rolesrCrolers r(r4z'GkeNodePoolTargetsParser._ValidateRoless  J**$ : 33G$NP P ..  11OOGj  / /   !r*c|syd}|D]E}|j|jj}||}+||k7s1tjddy)z7Validates that all pools specify an identical location.Nr z(All pools must have identical locations.)nodePoolConfigrrr@)r<initial_locationsrCrs r(r5z7GkeNodePoolTargetsParser._ValidatePoolsHaveSameLocationsi     (''11  $'   ) +33CE Er*cT|sytjd}|D]}|j 6|jj |jjjG|j |jjjrwt jddy)z>Validates that the bootDiskKmsKey matches the correct pattern.Nz>projects/[^/]+/locations/[^/]+/keyRings/[^/]+/cryptoKeys/[^/]+r z|bootDiskKmsKey must match pattern: projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME])recompilerLconfigrmatchrr@)r<boot_disk_kms_key_patternrCs r(r6z7GkeNodePoolTargetsParser._ValidateBootDiskKmsKeyPatterns  " H!J     ''..6!!((77? & , ,    $ $ 3 3511 h  r*NF) __name__ __module__ __qualname____doc__ staticmethodr2r3r4r5r6r*r(r/r/siO2, E E  r*r/ceZdZdZdddddZeddZe ddZedd Zed Z ed Z ed Z ed Z y)r1z@Helper to parse a --pools flag into a GkeNodePoolTarget message.rE CONTROLLER SPARK_DRIVERSPARK_EXECUTOR)r controllerz spark-driverzspark-executorc2tj||||S)aParses a --pools flag into a GkeNodePoolTarget message. Args: dataproc: The Dataproc API version to use for the GkeNodePoolTarget message. gke_cluster: The GKE cluster's relative name, for example, 'projects/p1/locations/l1/clusters/c1'. arg_pool: The dict[str, any] generated from the --pools flag. support_shuffle_service: support shuffle service. Returns: A GkeNodePoolTarget message. )r1_GkeNodePoolTargetFromArgPoolr7r8r;r:s r(r2z_GkeNodePoolTargetParser.Parses# $ A A+x)@ BBr*c|jjdj||dtj ||d|tj ||S)z3Creates a GkeNodePoolTarget from a --pool argument.z{0}/nodePools/{1}r r )r?r rL)rFrGformatr1 _SplitRoles_GkeNodePoolConfigFromArgPoolrbs r(raz6_GkeNodePoolTargetParser._GkeNodePoolTargetFromArgPool,se    . .$++K&9IJ&228Xg=N3JL/MM h / !!r*cg}tj}|r1tjj}|jddi|j dD]r}|j vrt jdd|z|j|jjj||j t|S)zFSplits the role string given as an argument into a list of Role enums.zshuffle-serviceSHUFFLE_SERVICE;r zUnrecognized role "%s".) r1_ARG_ROLE_TO_API_ROLEcopyupdatesplitlowerrr@rrFrGrH)r7 arg_rolesr:r defined_rolesarg_roles r(rez$_GkeNodePoolTargetParser._SplitRoles9s E6LL.DDIIKm-/@ABOOC(   .11 08;= = ll    - - K KHNN,- /0 ) Lr*c|jjtj||tj ||}d|vr|dj d|_||jjk7r|Sy)zECreates the GkeNodePoolConfig via the arguments specified in --pools.)rQ autoscalingrriN)rFGkeNodePoolConfigr1_GkeNodeConfigFromArgPool(_GkeNodePoolAutoscalingConfigFromArgPoolrmrr7r;rQs r(rfz6_GkeNodePoolTargetParser._GkeNodePoolConfigFromArgPoolJs   0 0'AA h , 1 1(H E 1GF h!+.44S9f ""4466 m r*c|jj}d|vr |d|_d|vr |d|_d|vr |d|_d|vr$|jj |d|_d|vrtj||d|_ d|vr |d|_ d|vr |d|_ ||jjk7r|Sy ) zACreates the GkeNodeConfig via the arguments specified in --pools.r r rr,)rrrrN) rF GkeNodeConfigr r rGkeEphemeralStorageConfigephemeralStorageConfigr1(_GkeNodePoolAcceleratorConfigFromArgPoolrrr)r7r;rCs r(ruz2_GkeNodePoolTargetParser._GkeNodeConfigFromArgPoolXs    * * ,D !-0d !-0d("#O4dh&$,$5$5$O$O !45%P%7d!!2[[ H^,.d8#$%56d8#$%56d x  ..00 k r*cVg}|jdD]l}d|vrtjdd|z|jdd\}} t|}|j |j j ||n|S#t$rtjdd|zwxYw)zPCreates the GkeNodePoolAcceleratorConfig via the arguments specified in --pools.ri=r zaaccelerators value "%s" does not match the expected "ACCELERATOR_TYPE=ACCELERATOR_VALUE" pattern.)acceleratorCountacceleratorTypez6Unable to parse accelerators count "%s" as an integer.)rmrr@r%rrFGkeNodePoolAcceleratorConfig ValueError)r7arg_acceleratorsrarg_acceleratoraccelerator_type count_stringcounts r(r|zA_GkeNodePoolTargetParser._GkeNodePoolAcceleratorConfigFromArgPoolrsL+11#6 O #11 <>MNO O(7'<'rsC<&' /.D NH V` ` FGGr*