gSXdZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZddlZdd lmZd ZGddej4ZGddeZGddeZGddeZGddeZGddeZ GddeZ!GddeZ"dZ#dZ$ejJd Z&Gd!d"ejNejPe)Z*Gd#d$e*Z+Gd%d&e*Z,d'Z-d(Z.d)Z/d*Z0d-d+Z1d.d,Z2y)/zUtilities for job submission preparation. The main entry point is UploadPythonPackages, which takes in parameters derived from the command line arguments and returns a list of URLs to be given to the AI Platform API. See its docstring for details. )absolute_import)division)unicode_literalsN) storage_util)uploads) exceptions)execution_utils)log)files)mapzfrom setuptools import setup, find_packages if __name__ == '__main__': setup( name='{package_name}', packages=find_packages(include=['{package_name}']) ) ceZdZdZy)UploadFailureErrorz0Generic error with the packaging/upload process.N__name__ __module__ __qualname____doc__5lib/googlecloudsdk/command_lib/ml_engine/jobs_prep.pyrr7s8rrc"eZdZdZfdZxZS)SetuptoolsFailedErrorz/Error indicating that setuptools itself failed.chdj|}|r|dz }n|dz }tt||y)Nz8Packaging of user Python code failed with message: {} zTTry manually writing a setup.py file at your package root and rerunning the command.zTry manually building your Python code by running: $ python setup.py sdist and providing the output via the `--packages` flag (for example, `--packages dist/package.tar.gz,dist/package2.whl)`)formatsuperr__init__)selfoutput generatedmsg __class__s rrzSetuptoolsFailedError.__init__?sM VF^ '(c MNc /4rrrrrr __classcell__r!s@rrr<s7 5 5rrc"eZdZdZfdZxZS)SysExecutableMissingErrorz/Error indicating that sys.executable was empty.cRtt| tjdy)Nz No Python executable found on path. A Python executable with setuptools installed on the PYTHONPATH is required for building AI Platform training jobs. )rr&rtextwrapdedent)rr!s rrz"SysExecutableMissingError.__init__Ps$ #T3  rr"r$s@rr&r&Ms7rr&c"eZdZdZfdZxZS)MissingInitErrorzCError indicating that the package to build had no __init__.py file.cptt| tjdj |y)Nz [{}] is not a valid Python package because it does not contain an `__init__.py` file. Please create one and try again. Also, please ensure that --package-path refers to a local directory. )rr+rr(r)r)r package_dirr!s rrzMissingInitError.__init__[s/ D*8??< , VK "rr"r$s@rr+r+XsK""rr+ceZdZdZy)UncopyablePackageErrorzError with copying the package.Nrrrrr/r/cs'rr/c"eZdZdZfdZxZS)DuplicateEntriesErrorzFError indicating that multiple files with the same name were provided.chtt| djdj |y)Nzrr7c"eZdZdZfdZxZS)InvalidSourceDirErrorz6Error indicating that the source directory is invalid.cJtt| dj|y)Nz/Source directory [{}] is not a valid directory.)rr9rr)r source_dirr!s rrzInvalidSourceDirError.__init__ws" /9@@LNrr"r$s@rr9r9ts>NNrr9c tjj|s t| t j |}|r|St j||rtdj||tjj|d}tjd|| t j|||S#t $r t|wxYw#t$rtdj|wxYw)arReturns a writable directory with the same contents as source_dir. If source_dir is writable, it is used. Otherwise, a directory 'dest' inside of temp_dir is used. Args: source_dir: str, the directory to (potentially) copy temp_dir: str, the path to a writable temporary directory in which to store any copied code. Returns: str, the path to a writable directory with the same contents as source_dir (i.e. source_dir, if it's writable, or a copy otherwise). Raises: UploadFailureError: if the command exits non-zero. InvalidSourceDirError: if the source directory is not valid. zVCannot copy directory since working directory [{}] is inside of source directory [{}].destz+Copying local source tree from [%s] to [%s]z%Cannot write to working location [{}])ospathisdirr9r HasWriteAccessInDir ValueErrorIsDirAncestorOfr/rr4r debugCopyTreeOSError)r;temp_dirwritabledest_dirs r_CopyIfNotWritablerJ|s& z "  ++,((4H  :x0  !!'*!= ??WW\\(F +()) 9:xPB NN:x( /% ,  ++, B /66x@ BBBsC.CC$Dc tjd|tjj |rtj d|yt j|}tj d|tj||y)aZGenerates a temporary setup.py file if there is none at the given path. Args: setup_py_path: str, a path to the expected setup.py location. package_name: str, the name of the Python package for which to write a setup.py file (used in the generated file contents). Returns: bool, whether the setup.py file was generated. z!Looking for setup.py file at [%s]z$Using existing setup.py file at [%s]F) package_namez&Generating temporary setup.py file: %sT) r rDr>r?isfileinfoDEFAULT_SETUP_FILErr WriteFileContents) setup_py_pathrLsetup_contentss r_GenerateSetupPyIfNeededrSsj)) /?WW^^M"HH 3]C %,,,,G.(( 4nE-8 rc# K tj}|j} ||r"|jt j yy#t$rd}|}Y9wxYw#|r"|jt j wwxYww)aYields a temporary directory or a backup temporary directory. Prefers creating a temporary directory (which will be cleaned up when the context manager is closed), but falls back to default_dir. There are systems where users can't write to temp, but we still need to copy. Args: default_dir: str, the backup temporary directory. Yields: str, the temporary directory. N)r TemporaryDirectory __enter__rF__exit__sysexc_info) default_dirrGr?s r_TempDirOrBackupr[s '')H    D ) Jh( H Dh(s8B$AA%%B A"B!A""B%&B  BcdeZdZdZdZej dZej dZdZ y)_SetupPyCommandaA command to run setup.py in a given environment. Includes the Python version to use and the arguments with which to run setup.py. Attributes: setup_py_path: str, the path to the setup.py file setup_py_args: list of str, the arguments with which to call setup.py package_root: str, path to the directory containing the package to build (must be writable, or setuptools will fail) c.||_||_||_yN)rQ setup_py_args package_root)rrQr`ras rrz_SetupPyCommand.__init__s&D&D$Drct)zEReturns arguments to use for execution (including Python executable).NotImplementedErrorrs rGetArgsz_SetupPyCommand.GetArgs   rct)z?Returns the environment dictionary to use for Python execution.rcres rGetEnvz_SetupPyCommand.GetEnvrgrc tj|jd|j|j|j|j S)zRun the configured setup.py command. Args: out: a stream to which the command output should be written. Returns: int, the return code of the command. T)no_exitout_funcerr_funccwdenv)r Execrfwriterari)routs rExecutez_SetupPyCommand.Executes@    syy399   4;;= 22rN) rrrrrabcabstractmethodrfrirsrrrr]r]sE %      2rr]ceZdZdZdZdZy)_CloudSdkPythonSetupPyCommandzA command that uses the Cloud SDK Python environment. It uses the same OS environment, plus the same PYTHONPATH. This is preferred, since it's more controlled. cltj|jg|jdt iS)Npython)r ArgsForPythonToolrQr`GetPythonExecutableres rrfz%_CloudSdkPythonSetupPyCommand.GetArgss:  , ,T-?-? K.2.@.@ K4G4I KKrctjj}tjj t j |d<|S)N PYTHONPATH)r>environcopypathsepr4rXr?)rexec_envs rriz$_CloudSdkPythonSetupPyCommand.GetEnvs2zz HZZ__SXX6H\ OrNrrrrrfrirrrrwrw sK rrwceZdZdZdZdZy)_SystemPythonSetupPyCommandzA command that uses the system Python environment. Uses the same executable as the Cloud SDK. Important in case of e.g. a setup.py file that has non-stdlib dependencies. cHt|jg|jzSr_)r{rQr`res rrfz#_SystemPythonSetupPyCommand.GetArgs's!  !4#5#5 69K9K KKrcyr_rres rriz"_SystemPythonSetupPyCommand.GetEnv*s rNrrrrrrsLrrcbd} tj}|S#t$r twxYwr_)r r{rBr&)python_executables rr{r{.s>&';;=  & # %%&s.c nt|5}dd|g}dd|d|g}dd|g}||z|z||z|f}g}|D]:} |jt|| ||jt|| |<|D]+} t j } | j | } | r+nt j dddtj|D cgc]"} tjj|| $}} tjd d j||S#1swYpxYwcc} w) aExecutes the setuptools `sdist` command. Specifically, runs `python setup.py sdist` (with the full path to `setup.py` given by setup_py_path) with arguments to put the final output in output_dir and all possible temporary files in a temporary directory. package_root is used as the working directory. May attempt to run setup.py multiple times with different environments/commands if any execution fails: 1. Using the Cloud SDK Python environment, with a full setuptools invocation (`egg_info`, `build`, and `sdist`). 2. Using the system Python environment, with a full setuptools invocation (`egg_info`, `build`, and `sdist`). 3. Using the Cloud SDK Python environment, with an intermediate setuptools invocation (`build` and `sdist`). 4. Using the system Python environment, with an intermediate setuptools invocation (`build` and `sdist`). 5. Using the Cloud SDK Python environment, with a simple setuptools invocation which will also work for plain distutils-based setup.py (just `sdist`). 6. Using the system Python environment, with a simple setuptools invocation which will also work for plain distutils-based setup.py (just `sdist`). The reason for this order is that it prefers first the setup.py invocations which leave the fewest files on disk. Then, we prefer the Cloud SDK execution environment as it will be the most stable. package_root must be writable, or setuptools will fail (there are temporary files from setuptools that get put in the CWD). Args: package_root: str, the directory containing the package (that is, the *parent* of the package itself). setup_py_path: str, the path to the `setup.py` file to execute. output_dir: str, path to a directory in which the built packages should be created. Returns: list of str, the full paths to the generated packages. Raises: SysExecutableMissingError: if sys.executable is None RuntimeError: if the execution of setuptools exited non-zero. sdistz --dist-dirbuildz --build-basez --build-tempegg_infoz --egg-baseNz!Python packaging resulted in [%s]r3)r[appendrwrioStringIOrs RuntimeErrorgetvaluer>listdirr?r4r rD)rarQ output_dir working_dir sdist_args build_args egg_info_argssetup_py_arg_setssetup_py_commandsr`setup_py_commandrr return_coderel_file local_pathss r_RunSetupToolsr7s_d %<4J nkKJ {;M "Z/Z * <   67:   67+ . KKMc$,,S1k  .  (( ;&D"$J!79!7Xj(3!79)) /;1GH I&%B9sB D&D&'D2&D/c tjj|}tjj|}t |5}t ||}tjj tjj|ds t|tjj|d}tjj|}t||} t||||r?tjj|d}||fD]} tj|cdddS#t$rtjd|YFwxYw#t $r$} t#t%j&| |d} ~ wwxYw#|retjj|d}||fD]=} tj|#t$rtjd|Y;wxYwwwxYw#1swYyxYw)azBuilds Python packages from the given package source. That is, builds Python packages from the code in package_path, using its parent directory (the 'package root') as its context using the setuptools `sdist` command. If there is a `setup.py` file in the package root, use that. Otherwise, use a simple, temporary one made for this package. We try to be as unobstrustive as possible (see _RunSetupTools for details): - setuptools writes some files to the package root--we move as many temporary generated files out of the package root as possible - the final output gets written to output_dir - any temporary setup.py file is written outside of the package root. - if the current directory isn't writable, we silenly make a temporary copy Args: package_path: str. Path to the package. This should be the path to the directory containing the Python code to be built, *not* its parent (which optionally contains setup.py and other metadata). output_dir: str, path to a long-lived directory in which the built packages should be created. Returns: list of str. The full local path to all built Python packages. Raises: SetuptoolsFailedError: If the setup.py file fails to successfully build. MissingInitError: If the package doesn't contain an `__init__.py` file. InvalidSourceDirError: if the source directory is not valid. z __init__.pyzsetup.pyz setup.pycz;Couldn't remove file [%s] (it may never have been created).N)r>r?abspathdirnamer[rJexistsr4r+basenamerSrunlinkrFr rDrrsix text_type) package_pathrrarrQrLrpyc_filer?errs r BuildPackagesrsB.,., %%lK@L 77>>"'',,|]C D \ **GGLLz:M77##L1L( EI L- D  77<< k:"H-D  IIdO.3&%8  IIM   A !#--"4i @@A  77<< k:"H-D  IIdO  IIM  . '&%s B H+ E'7)H!E6HE$ !H#E$ $H' F0FFF*HGHG: 7H9G: :HHH c|s ttjtt t j j|}tj|Dcgc] \}}|dkDs |}}}|r t||Dcgc]#}|t j j|f%}}tj||j|jScc}}wcc}w)z;Uploads files after validating and transforming input type.)r7 collectionsCounterlistr r>r?rr iteritemsr1r UploadFiles bucket_refname)pathsstaging_locationcounterrcountr5r? upload_pairss r_UploadFilesByPathrs  ""   S)9)95%A B C'(+ g(>L(>u%!)(>*L  ++=BCUT4))$/0U,C   \+;+F+F-22 44 MDs% C 3C  (C&c Dg}g}|D]D}tjj|r|j|4|j|F|rtj j tj j|}t|5}|jt|tj j|d|jt||ddd|S|r|jt|||S#1swY|SxYw)aUploads Python packages (if necessary), building them as-specified. An AI Platform job needs one or more Python packages to run. These Python packages can be specified in one of three ways: 1. As a path to a local, pre-built Python package file. 2. As a path to a Cloud Storage-hosted, pre-built Python package file (paths beginning with 'gs://'). 3. As a local Python source tree (the `--package-path` flag). In case 1, we upload the local files to Cloud Storage[1] and provide their paths. These can then be given to the AI Platform API, which can fetch these files. In case 2, we don't need to do anything. We can just send these paths directly to the AI Platform API. In case 3, we perform a build using setuptools[2], and upload the resulting artifacts to Cloud Storage[1]. The paths to these artifacts can be given to the AI Platform API. See the `BuildPackages` method. These methods of specifying Python packages may be combined. [1] Uploads are to a specially-prefixed location in a user-provided Cloud Storage staging bucket. If the user provides bucket `gs://my-bucket/`, a file `package.tar.gz` is uploaded to `gs://my-bucket///package.tar.gz`. [2] setuptools must be installed on the local user system. Args: packages: list of str. Path to extra tar.gz packages to upload, if any. If empty, a package_path must be provided. package_path: str. Relative path to source directory to be built, if any. If omitted, one or more packages must be provided. staging_location: storage_util.ObjectReference. Cloud Storage prefix to which archives are uploaded. Not necessary if only remote packages are given. Returns: list of str. Fully qualified Cloud Storage URLs (`gs://..`) from uploaded packages. Raises: ValueError: If packages is empty, and building package_path produces no tar archives. SetuptoolsFailedError: If the setup.py file fails to successfully build. MissingInitError: If the package doesn't contain an `__init__.py` file. DuplicateEntriesError: If multiple files with the same name were provided. ArgumentError: if no packages were found in the given path or no staging_location was but uploads were required. rN) rObjectReference IsStorageUrlrr>r?rrr[extendrr4r)packagesrr remote_pathsrpackagerars rUploadPythonPackagesrsl,+g##009'"!  77??277??<#@AL , ';|')ww||K'JLM,[:JKL (  *;8HIJ  ( s ADDc (d}|r"tjj||}|S|rdtjj|jdj |j jddfDcgc]}|r|c}}|Scc}w)zEGet the appropriate staging location for the job given the arguments.N/r)rr FromBucketRefFromNamebucketr4rrstrip)job_idstaging_bucketjob_dirrfs rGetStagingLocationr/s#33AA   #33<<gll.A.A#.F.8.:"@.:=>#$.:"@AB "@s7 B)rNN)NNN)3r __future__rrrrtr contextlibrr>rXr(googlecloudsdk.api_lib.storager$googlecloudsdk.command_lib.ml_enginergooglecloudsdk.corerr r googlecloudsdk.core.utilr r six.movesr rOErrorrrr&r+r/r1r7r9rJrScontextmanagerr[with_metaclassABCMetaobjectr]rwrr{rrrrrrrrrs<''  78*/#* )) 5.5" 2")"(/($.$?/?N.N+\, ))@(2(c((f=(2VO& / VrBJ 4IX r