֏dZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZddl mZddl mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlm Z ddl!m"Z#ddl!m$Z%ddl&m'Z'ddl(Z(ejRjUejRjWe,dZ-dZ. d6dZ/dZ0Gdde1Z2dZ3dZ4d7dZ5dZ6 d8dZ7Gd d!e1Z8 d9d"Z9d#Z:d$Z;d%Zd(Z?d)Z@d*ZAd:d+ZBd,ZCd-ZDd.ZEd/ZFGd0d1e1ZGd:d2ZHd3ZId4ZJd5ZKy);z.Common utilities for the gcloud dataproc tool.)absolute_import)division)unicode_literalsN)encoding) exceptions) list_pager)storage_helpers) arg_parsers)util)log) properties)requests) console_attr) console_io)progress_tracker)creds)store)retryschemascptjdtj|z|jS)zReturns a printable representation of a failed Google API's status.proto. Args: error: the failed Status to print. Returns: A ready-to-print string representation of the error. zError: )r debugr MessageToJsonmessageerrors +lib/googlecloudsdk/api_lib/dataproc/util.pyFormatRpcErrorr3s*))J//6 67 c.tj|d5tj}|tj|z kDr9 ||tj||tj|z kDr9dddtjdj|#tj$r Ydddytj $r-}t jd||t|rYd}~d}~wwxYw#1swYxYw)z1Poll Dataproc resource until it no longer exists.TautotickNz!Get request for [{0}] failed: {1}z"Deleting resource [{0}] timed out.) rProgressTrackertimeapitools_exceptionsHttpNotFoundError HttpErrorr rIsClientHttpExceptionsleeprOperationTimeoutErrorformat)request_method resource_refr timeout_s poll_period_s start_timers rWaitForResourceDeletionr0@s ''$?J tyy{Z/ 0 |$ jj tyy{Z/ 0@ ((*11,? AA! 2 2@?! * * 6 eL ! '  ( @?sA/D B,/D ,D?D  D#D>D DD  Dc>tjjSN)uuiduuid4hexrr GetUniqueIdr7Ys   rceZdZdZdZy)Bunchz}Class that converts a dictionary to javascript like object. For example: Bunch({'a': {'b': {'c': 0}}}).a.b.c == 0 ctj|D]/\}}t|tr t |}||j |<1yr2)six iteritems isinstancedictr9__dict__)self dictionarykeyvalues r__init__zBunch.__init__ds:mmJ/ U E4 e  dmmC0rN)__name__ __module__ __qualname____doc__rDr6rrr9r9]s  !rr9cT|jddd|jdddy)Nz--jarmain_jarz3The HCFS URI of jar file containing the driver jar.)desthelpz--class main_classzuThe class containing the main method of the driver. Must be in a provided jar or jar that is already on the classpath) add_argument)parsers rAddJvmDriverFlagsrPksA  @B   DFrcB|jdk\xr|jdkS)z>Returns true if the http exception given is an HTTP 4xx error.ii) status_code)http_exceptions rr'r'ws#  # #s * O~/I/IC/OOrc| |jj|j}tjj dj |jtj}d tjd}|rdnd fd}tj|d 5|tj|z kDr |jjj|}t|j |jj"} || j$t'| j$ |j(rn1 tj0||tj|z kDrd d d t|j |jj"} || j$|j(s.t3j4d j |j|j6rBt3j8d j |jt;|j6tj<d |jtj|z |S#t*j,$r} t/| rYd } ~ Yd } ~ wwxYw#1swY3xYw)aIPoll dataproc Operation until its status is done or timeout reached. Args: dataproc: wrapper for Dataproc messages, resources, and client operation: Operation, message of the operation to be polled. message: str, message to display to user while polling. timeout_s: number, seconds to poll with retries before timing out. poll_period_s: number, delay in seconds between requests. Returns: Operation: the return value of the last successful operations.get request. Raises: OperationError: if the operation times out or finishes with an error. nameWaiting on operation [{0}].rTr c|d}|r._LogWarningss?O,-L ggmm%&!' G"rr NOperation [{0}] timed out.Operation [{0}] failed: {1}.*Operation [%s] finished after %.3f seconds)messages+DataprocProjectsRegionsOperationsGetRequestrVr statusPrintr*r#r IsInteractiverr"clientprojects_regions_operationsGetParseOperationJsonMetadatametadataClusterOperationMetadatar^lendoner$r&r'r(rr)rOperationErrorrinfo) dataproc operationrr-r.requestr/is_ttyrbrorSr`ras @@rWaitForOperationry}s-"    I I >> J '**077 GHyy{*/  # #$ /&$d"''$? tyy{Z/ 0 OO??CCGL -    1 1 J JLX&&'h//0 >>    jj tyy{Z/ 0@((++DDF(x  !   * *+G+N+N,   # #$B$I$Iy7%9 ::(( 7 IIK* $' #! * *  0  1@?s70J1 BJ1J1J. J)#J1)J..J11J;cn d}||vs|j||k7r|jDtjj dj |j|jn8tjj dj |j|j||<|j |dk7rBt|j drJ|j j4tjj |j jnt|j drO|j j9tjj dj |jnnt|j d rX|j jBtjj d j |j j|j |d<t|jd rD|jjD]*}|js|j|vs||j|jk7r\tjj d j |j|j|j||j<|js|j|vs||j|jk7stjj d j |j|j|j||j<-|j|dk7rCt|jdrJ|jj4tjj |jjnt|jdrO|jj9tjj dj |jnnt|jd rX|jjBtjj dj |jj|j|d<yy)ahPrint workflow and job status for the running workflow template. This method will detect any changes of state in the latest metadata and print all the new states in a workflow template. For example: Workflow template template-name RUNNING Creating cluster: Operation ID create-id. Job ID job-id-1 RUNNING Job ID job-id-1 COMPLETED Deleting cluster: Operation ID delete-id. Workflow template template-name DONE Args: metadata: Dataproc WorkflowMetadata message object, contains the latest states of a workflow template. status: Dictionary, stores all jobs' status in the current workflow template, as well as the status of the overarching workflow. operations: Dictionary, stores cluster operation status for the workflow template. errors: Dictionary, stores errors from the current workflow template. wtNzWorkflowTemplate [{0}] {1}zWorkflowTemplate {0} createClusterrrrzCreated cluster: {0}. operationIdz%Creating cluster: Operation ID [{0}].nodeszJob ID {0} {1}zJob ID {0} error: {1} deleteClusterzDeleted cluster: {0}.z%Deleting cluster: Operation ID [{0}].)statetemplater rhrir*r|hasattrrrr clusterNamer}graphr~jobIdr)rorh operationserrors template_keynodes rPrintWorkflowMetadatars0,8>>VL5I#I$ jj3::   X^^-. jj-44X^^DE#>>F< z/::x%%$2288D jjx--334 '' %3388D jj.55h6J6JKL  #11==I jj>EE  , ,./"*"8"8J X^^W%$$ ZZ 6 !VDJJ%74::%E )00TZZHI!ZZtzz 61 +tzz9 077 DJJOP!ZZtzz%z/::x%%$2288D jjx--334 '' %3388D jj.55h6J6JKL  #11==I jj>EE  , ,./"*"8"8J;rc\|jj|j}tjj dj |jtj}ddd}i}i}||tj|z kDr |jjj|}t|j|jj} t| ||||jrn4 tj&|||tj|z kDrt|j|jj} |js.t)j*dj |j|j,rBt)j.dj |jt1|j,dD][} | |vs|| || j,st)j.dj || j2|| j,tj4d|jtj|z |S#t j"$r} t%| rYd} ~ d} ~ wwxYw)a Poll dataproc Operation until its status is done or timeout reached. Args: dataproc: wrapper for Dataproc messages, resources, and client operation: Operation, message of the operation to be polled. timeout_s: number, seconds to poll with retries before timing out. poll_period_s: number, delay in seconds between requests. Returns: Operation: the return value of the last successful operations.get request. Raises: OperationError: if the operation times out or finishes with an error. rUrWN)r|rrcrdre)rfrgrVr rhrir*r#rkrlrmrnroWorkflowMetadatarrrr$r&r'r(rr)rrsrr}rt) rurvr-r.rwr/rrhrrorSops r WaitForWorkflowTemplateOperationrsM&    I I >> J '**077 GHyy{*!%=* & & Y$))+ *BC //==AA'Ji+I,>,>,4,=,=,N,NPhHfj&A     JJ} Y$))+ *BC( (:(:(0(9(9(J(JL(   * *+G+N+N,   # #$B$I$Iy7%9 :: .b ZJrN6:b>;O;O  % %&D&K&K R. $ $jn&:&:'< == / (( 7 IIK* $' +  ( ( ~ .  /sA)JJ+ J&&J+ceZdZdZdZdZy)NoOpProgressDisplayz8For use in place of a ProgressTracker in a 'with' block.cyr2r6r@s r __enter__zNoOpProgressDisplay.__enter__Ercyr2r6)r@ unused_argss r__exit__zNoOpProgressDisplay.__exit__HrrN)rErFrGrHrrr6rrrrBs@  rrc |jj|j|j|j} dd} d} d} d}fd}d}|r*t j jdt} ntj|d } tjx}}| 5| r | ||z kDr:|xr j }| s,|j j|jvr d} ||z d z} | r|r|rn| xr|| |zk\}|xr }| xr|}|s|s|r|} |jj j#| }|rk|j2r_|j2|k7rP|r#|t j(d ||j2}t5j6|j2tj8|tj}| s0| ||z kDr:ddd|j j}||jvr@|r9st j(d n!jrt j(d||ur|S|r||ur|j j:rCt=j>dj+|j|j j:t=j>dj+|j|j j:r,t j@d|j j:zt=j>dj+|j||t=jBdj+|j|#t$j&$rN}t j(d j+t-j.|t1|rYd}~d}~wwxYw#1swYxYw)aPoll dataproc Job until its status is terminal or timeout reached. Args: dataproc: wrapper for dataproc resources, client and messages job: The job to wait to finish. job_ref: Parsed dataproc.projects.regions.jobs resource containing a projectId, region, and jobId. message: str, message to display to user while polling. goal_state: JobStatus.StateValueValuesEnum, the state to define success error_state: JobStatus.StateValueValuesEnum, the state to define failure stream_driver_log: bool, Whether to show the Job's driver's output. log_poll_period_s: number, delay in seconds between checking on the log. dataproc_poll_period_s: number, delay in seconds between requests to the Dataproc API. timeout_s: number, time out for job completion. None means no timeout. Returns: Job: the return value of the last successful jobs.get request. Raises: JobError: if the job finishes with an error. ) projectIdregionrNrFcdr-jr jtjyyyr2)openReadIntoWritabler r[)driver_log_streamsrReadDriverLogIfPresentz5WaitForJobTermination..ReadDriverLogIfPresentts).33((14rctj}tjj d|j dzy)N=r)rGetConsoleAttrr r[ri GetTermSize)attrs rPrintEqualsLinez.WaitForJobTermination..PrintEqualsLineys3  & & (DGGMM#((*1--.rzWaiting for job output...Tr zGetJob failed: {}z3Job attempt failed. Streaming new attempt's output.zExpected job output not found.z4Job terminated, but output did not finish streaming.z Job [{0}] failed with error: {1}zJob [{0}] failed.z Details: z6Job [{0}] entered state [{1}] while waiting for [{2}].z)Job [{0}] timed out while in state [{1}].)"rf%DataprocProjectsRegionsJobsGetRequestrrrr rhrirrr"r#rrterminal_job_statesrkprojects_regions_jobsrmr$r&r]r*r; text_typer'driverOutputResourceUrir StorageObjectSeriesStreamr(detailsrJobErrorrtJobTimeoutError)rujobjob_refr goal_state error_statestream_driver_loglog_poll_period_sdataproc_poll_period_sr-rwlast_job_poll_time job_complete wait_displaydriver_output_urirrr/nowlog_stream_closedregular_job_pollexpecting_output_streamexpecting_job_donerrrs @rWaitForJobTerminationrLs@    C C!!'..  D O',,2 /JJ01&(L#33GdKLYY[ *s9j(89+J4E4J4J0J **  h:: : *$r) 04E   A'*@@@ !2 K:K6K++A0A 48J  5599'B# #"="=  ' '+< <    KKM N  !99 -GG))+  jj"# IIKcS9j(89X **  % h***  45  ! ! JK  ju +   !!"E"L"L MM3::--#/0 0    3 : :7== I JJ zz hh|cjj0001   @GG MM5* . // ""188N PPM#,,  ++*11#--2FG H "5 ) * 7|s?+A>@/  1;- @ A'**,11W724##s* 4 55jj+334) ~t ,,rctjddd}tj|tj|r |j }n |j }|stjdt||S)a<Get an access token for the user's current credentials. Args: access_boundary_json: JSON string holding the definition of the access boundary to apply to the credentials. Raises: PersonalAuthError: If no access token could be fetched for the user. Returns: An access token for the user. NT)allow_account_impersonationuse_google_authz?No access token could be obtained from the current credentials.) c_storeLoadRefreshc_credsIsOauth2ClientCredentialsrr$rPersonalAuthErrorr*)r%credr$s rGetCredentialsr5sq  d D$ //$ &&t,   E JJE   & &I KK u&: ;;rcheZdZdZdZd dZdZdZejdd Z d Z d Z y) PersonalAuthUtilsz2Util functions for enabling personal auth session.cyr2r6rs rrDzPersonalAuthUtils.__init__rrNc|g}|j|d} ttddr{tj||tjtjd}|j j dj}|j|jStj|tjtjtj}|j|\}} |S#t$rm} |r&tjdd j||t!j"d d j|zd zt%j&| zd} ~ wwxYw) aiRun the specified command, capturing and returning output as appropriate. Args: openssl_executable: The path to the openssl executable. args: The arguments to the openssl command to run. stdin: The input to the command. Returns: The output of the command. Raises: PersonalAuthError: If the call to openssl fails NrunF)inputstdoutstderrcheckutf-8)stdinr<r=)r;z3OpenSSL command "%s" failed with error message "%s" z"Failure running openssl command: "z": )extendgetattr subprocessr:PIPEr=decodestripcheck_returncoder<Popen communicate Exceptionr rr rr3r;r) r@openssl_executableargsr@commandr=procpr<_exs r_RunOpensslCommandz$PersonalAuthUtils._RunOpensslCommands>""G NN4 F!< UD )~~ ????  ##G,224 {{    //????  $ MMM.  <  G((7#V -  ( ()M),'):*;=B*C),r):*; << $AC>> E4A(E//E4c|j|ddd|g|jd} |jjdd}t |dk7r t d t |d |d zjdS#t$r0}tjd tj|zd }~wwxYw)zCompute HMAC tag using OpenSSL.dgstz-sha256z-hmacr@r?rA@z1HMAC output is expected to be 64 characters long.z'Failure due to invalid openssl output: NrX) rSrFrGr rqrintrKrr3r;rencode)r@rBrrL cmd_outputstripped_outputrRs r _ComputeHmaczPersonalAuthUtils._ComputeHmacs((VY=)F7OI"((*005a8o _  #LMM /2 d " * *7 33 I  ( ( 3cmmB6G G IIIsAB B= +B88B=c"t|dk7r tdtj|j d}|j |d|}t |}|j||jd|j |||S)a}Derives HMAC-based Key Derivation Function (HKDF) key through expansion on the initial pseudorandom key. Args: prk: a pseudorandom key. info: optional context and application specific information (can be empty). openssl_executable: The path to the openssl executable. Returns: Output keying material, expected to be of 256-bit length. zCThe given initial pseudorandom key is expected to be 32 bytes long.r?r)rqrbase64 b16encoderFr^ bytearrayrB)r@prkrtrL base16_prkt1t2datas r_DeriveHkdfKeyz PersonalAuthUtils._DeriveHkdfKeys 3x2~  O QQ!!#&--g6J  :s,> ?B r]F MM$ MM'   Z1C DDrrW) max_retrialsc tj|dzjdj}t j t jd}|jd}t jd}|j|djd|}t j |j|djd|jd} tj5} | j|jd| jd|j|dd d d d | jgt j | } dddt# dk7r t%dt j | jd} tj5} | j|| jddddd|ddj'| jg}|j|||jd }dddt#| dk7r t%dt j jd}t)|}|j+||j-| ||jddd}dj'||| ||S#1swY^xYw#1swYxYw)zEncode token using OpenSSL. Args: public_key: The public key for the session/cluster. secret: Token to be encrypted. openssl_executable: The path to the openssl executable. Returns: Encrypted token. rXr?rYr`encryption_keyauth_keyrrsautlz-oaepz-encryptz-pubinz-inkeyrVNiz3The encrypted key is expected to be 512 bytes long.encz -aes-256-ctrz-saltz-ivz-passzfile:{}z{}:{}:{}:{}:{})hashlibsha256r[ hexdigestrbrcosurandomrFritempfileNamedTemporaryFiler\seekrSrV b64encoderqrr*rdrBr^)r@ public_keysecretrLkey_hashiv_bytesinitialization_vector initial_keyrlrmkf encrypted_key encoded_keypf encrypt_argsencrypted_token encoded_token hmac_inputhmac_tags r_EncodeTokenUsingOpensslz*PersonalAuthUtils._EncodeTokenUsingOpenssls~~zD088ABLLNH 2/H$OOG4**R.K(()9)@)@)I);=N K):):7)C. 0117   $ $ &"hhz  )*ggaj--  Wj(Hbgg F  -./m ' =S L MM""=188AK  $ $ &"hh~ggaj %1F   277 #l// l&--2H0Jo ' =S L MM$$_5<A)KA(KK Kc~|r|j|||S ddl}ddlm}|j d}tj|dddd}tj|d zjd j}|j|} |j| } | j|j } | j#|jd |} t%j&| j)d } d j+|| S#t$rt j dwxYw) zEncrypt secret with resource public key. Args: public_key: The public key for the session/cluster. secret: Token to be encrypted. openssl_executable: The path to the openssl executable. Returns: Encrypted token. rNhybridaCCannot load the Tink cryptography library. Either the library is not installed, or site packages are not enabled for the Google Cloud SDK. Please consult Cloud Dataproc Personal Auth documentation on adding Tink to Google Cloud SDK for further instructions. https://cloud.google.com/dataproc/docs/concepts/iam/personal-authrrBkeyDatarCrXr?z{}:{})rtinkr ImportErrorrr3registerr r!rprqr[rrJsonKeysetReaderread_no_secret_keyset_handle primitive HybridEncryptencryptrbrxrFr*)r@ryrzrLrrcontextpublic_key_valuer{readerkh_pub encrypter ciphertextrs rEncryptWithPublicKeyz&PersonalAuthUtils.EncryptWithPublicKeyGs=  * *:v+=?? O OOGzz*-e4Q7 B7K~~ D ((133<9;  " ": .F  . .v 6F  !5!56I""6==#97CJ$$Z077@M >>(M 223 O  ( ( N OOOs DD<c8 ddl}ddlm}y#t$rYywxYw)z1Check if Tink cryptography library can be loaded.rNrTF)rrr)r@rrs rIsTinkLibraryInstalledz(PersonalAuthUtils.IsTinkLibraryInstalledus&   s r2) rErFrGrHrDrSr^rirRetryOnExceptionrrrr6rrr7r7sJ: 2._GetFieldFns$w**+ 7D !!rget_field_funcz, N)setr YieldFromListr r]r sorted)unreachable_warning_msgrMkwargsrresultitemrs @r YieldFromListWithUnreachableListrsf+"  # # P; P P&d JKK &%&sA$A')<)r)Nr)NFrWrNr2)LrH __future__rrrrbrpr rsrDrur#r3apitools.base.pyrrr$rgooglecloudsdk.api_lib.dataprocr googlecloudsdk.callioper !googlecloudsdk.command_lib.exportr rgooglecloudsdk.corer r rgooglecloudsdk.core.consolerrrgooglecloudsdk.core.credentialsrr1rr.googlecloudsdk.core.utilrr;pathr dirname__file__ SCHEMA_DIRrr0r7objectr9rPr'ryrrrrrrrrrnrrrrrrr*r5r7rrrrr6rrrsU5&'   %>'6;/A#*(428<<* WW\\"''//(3Y ?   ')*+ A2 !F ! FP <~E9V0434:z & '+,1,-13$(yP~79  H   #L.$-8<4VVr$( r