dZddlmZddlmZddlmZddlZddlZddlmZddlm Z ddl m Z dd l m ZddlZdd lmZej$j&ZGd d eZGd de j.ZddZdZddZy)zCommon ML file upload logic.)absolute_import)division)unicode_literalsN) storage_api) storage_util) exceptions)files)zipceZdZdZy)MissingStagingBucketExceptionzIndicates that a staging bucket was not provided with a local path. It doesn't inherit from core.exceptions.Error because it should be caught and re-raised at the call site with an actionable message. N__name__ __module__ __qualname____doc__3lib/googlecloudsdk/command_lib/ml_engine/uploads.pyr r $srr ceZdZdZy)BadDirectoryErrorz9Indicates that a provided directory for upload was empty.Nr rrrrr,sArrc tjtj}|D]\}}|j ||"dj ||j g}n|j }tj}g}|D]\}}tjj|dj ||g} |j|| } |jdj d| j| jg|S)aUploads files at the local path to a specifically prefixed location. The prefix is 'cloudmldist/'. Args: upload_pairs: [(str, str)]. Pairs of absolute paths to local files to upload and corresponding path in Cloud Storage (that goes after the prefix). For example, ('/path/foo', 'bar') will upload '/path/foo' to '/bar' in Cloud Storage. bucket_ref: storage_util.BucketReference. Files will be uploaded to this bucket. gs_prefix: str. Prefix to the GCS Path where files will be uploaded. Returns: [str]. A list of fully qualified gcs paths for the uploaded files, in the same order they were provided. ) algorithm/zgs:/) file_utilsChecksumhashlibsha256AddFileContentsjoin HexDigestr StorageClientrObjectReference FromBucketRef CopyFileToGCSappendbucketname) upload_pairs bucket_ref gs_prefixchecksum local_path_storage_clientdests uploaded_pathobj_refobjs r UploadFilesr30s$ 7>> :(#mj! Z($)X%7%7%9:;I""$I,,.. %#/j-**88CHHi78:G  & &z7 ;C LL63::sxx89: $0 ,rcg}tjtj|D]]\}}}|D]R}tjj ||}|j tjj||T_|S)aeReturn all the descendents of root, relative to its path. For instance, given the following directory structure /path/to/root/a /path/to/root/a/b /path/to/root/c This function would return `['a', 'a/b', 'c']`. Args: root: str, the path to list descendents of. Returns: list of str, the paths in the given directory. )start)oswalksix text_typepathrr%relpath)rootpathsdirpathr- filenamesfilenameabs_paths r_GetFilesRelativerBUsn" %!wws}}T':;gq)gx0h ll277??84?89 < ,rc|jdr|S| tdtjj |st dj |t|}|Dcgc]}|jtd}}|Dcgc]}tj||g}}ttt||||}|st dj ||ddt|d Scc}wcc}w) aBUploads path to Cloud Storage if it isn't already there. Translates local file system paths to Cloud Storage-style paths (i.e. using the Unix path separator '/'). Args: path: str, the path to the directory. Can be a Cloud Storage ("gs://") path or a local filesystem path (no protocol). staging_bucket: storage_util.BucketReference or None. If the path is local, the bucket to which it should be uploaded. gs_prefix: str, prefix for the directory within the staging bucket. Returns: str, a Cloud Storage path where the directory has been uploaded (possibly prior to the execution of this function). Raises: MissingStagingBucketException: if `path` is a local path, but staging_bucket isn't found. BadDirectoryError: if the given directory couldn't be found or is empty. zgs://NzKThe path provided was local, but no staging bucket for upload was provided.z[{}] is not a valid directory.r)r*zYCannot upload contents of directory [{}] to Google Cloud Storage; directory has no files.r) startswithr r6r:isdirrformatrBreplace _PATH_SEPrr3listr len)r:staging_bucketr*r fr/ full_filesuploaded_pathss rUploadDirectoryIfNecessaryrOns, __W K '   t  <CCDI JJ D !%/4 4e199Y $e% 43895a ay)5*9tC E$:;-)24.   ""(&, 00  ?Sq]N ++ 59s )C> D)N)NN)r __future__rrrrr6googlecloudsdk.api_lib.storagerrgooglecloudsdk.corergooglecloudsdk.core.utilr rr8 six.movesr r:seprH Exceptionr Errorrr3rBrOrrrrXsj#&' 67*8  GGKK IB ((B"J24,r