dZddlmZddlmZddlmZddlmZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZddl mZdd l mZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddl m!Z!ddl"m#Z#ddl"m$Z$e jJre&Z'e jPdZ)dZ*Gdde+Z,Gdde,Z-dZ.Gdde,Z/Gd d!e'Z0 d#d"Z1y)$z1Wildcard iterator class and supporting functions.)absolute_import)print_function)division)unicode_literalsN)BucketListingBucket)BucketListingObject)BucketListingPrefix)AccessDeniedException)CloudApi)NotFoundException)CommandException)ContainsWildcard)GenerationFromUrlAndString)StorageUrlFromString) StripOneSlash)WILDCARD_REGEX)storage_v1_messages)UTF8)FixWindowsEncodingIfNeeded) PrintableStrz (?P.*?)\*\*(?P.*)a\Invalid Unicode path encountered (%s). gsutil cannot proceed with such files present. Please remove or rename this file and try again. NOTE: the path printed above replaces the problematic characters with a hex-encoded printable representation. For more details (including how to convert to a gsutil-compatible encoding) see `gsutil help encoding`.ceZdZdZdZy)WildcardIteratoraClass for iterating over Google Cloud Storage strings containing wildcards. The base class is abstract; you should instantiate using the wildcard_iterator() static factory method, which chooses the right implementation depending on the base string. c4d|jjzS)z2Returns string representation of WildcardIterator.zWildcardIterator(%s)) wildcard_url url_stringselfs *platform/gsutil/gslib/wildcard_iterator.py__repr__zWildcardIterator.__repr__Is !D$5$5$@$@ @@N)__name__ __module__ __qualname____doc__rr rrr?s Ar rcreZdZdZ ddZ ddZdZdZdZddZ dd Z dd Z d Z dd Z dd ZddZy)CloudWildcardIteratoraVWildcardIterator subclass for buckets, bucket subdirs and objects. Iterates over BucketListingRef matching the Url string wildcard. It's much more efficient to first get metadata that's available in the Bucket (for example to get the name and size of each object), because that information is available in the object list results. Ncv||_||_||_||_|xst j |_y)aInstantiates an iterator that matches the wildcard URL. Args: wildcard_url: CloudUrl that contains the wildcard to iterate. gsutil_api: Cloud storage interface. Passed in for thread safety, also settable for testing/mocking. all_versions: If true, the iterator yields all versions of objects matching the wildcard. If false, yields just the live object version. project_id: Project ID to use for bucket listings. logger: logging.Logger used for outputting debug messages during iteration. If None, the root logger will be used. N)r all_versions gsutil_api project_idlogging getLoggerlogger)rrr*r)r+r.s r__init__zCloudWildcardIterator.__init__Ws7&%D$D DO DO/G--/DKr c # K|jj}d}|rt}|D]}|j||j |}|j ddg|j dg|s |j r&|j ddg|j dd g|jd g D]2}|j}|jjr|rt|}|jj|jd |j |jj| D]r} | jt j"j$k(r+|j'|| j(|j U|j+|| j(t|t-|jjs|jj/r|j s |jj1|jj|jj2|jj4|jj|} |j'|jj6| |j xs|y|r2||jj2d|jj4} n'|t=|jj2xsd } | g} | sLt| j?d}|jA|j2\} }}}|jC|}|r tdgn|}|jj|j| ||j xs||jj|D]7} |D].}| jt j"j$k(r| j(}|jE|jFsS|rt=|jF|k(rA|r|jI|j4r$|j'|||j xs|| j(} t-| rtKd|| dt=| }|jE|s|r#||k7r||d z|z}| jM|n|j+|| 7:| r5y#t8t:f$rY_wxYww)a!Iterator that gets called when iterating over the cloud wildcard. In the case where no wildcard is present, returns a single matching object, single matching prefix, or one of each if both exist. Args: bucket_listing_fields: Iterable fields to include in bucket listings. Ex. ['name', 'acl']. Iterator is responsible for converting these to list-style format ['items/name', 'items/acl'] as well as adding any fields necessary for listing such as prefixes. API implementation is responsible for adding pagination fields. If this is None, all fields are returned. expand_top_level_buckets: If true, yield no BUCKET references. Instead, expand buckets into top-level objects and prefixes. Yields: BucketListingRef of type BUCKET, OBJECT or PREFIX. N) get_fieldsz items/nameprefixesnamezitems/generationzitems/metageneration generationmetagenerationid bucket_fields/) delimiterr)providerfields) with_version)r4r;r<#r)prefixr:r)r;r<z Cloud folder z] contains a wildcard; gsutil does not currently support objects with wildcards in their name.)'r HasGenerationsetadd_GetToListFieldsupdater)_ExpandBucketWildcardsrIsBucketrr* ListObjects bucket_nameschemedatatyper CsObjectOrPrefixTypeOBJECT _GetObjectRefdata _GetPrefixRefrIsObjectGetObjectMetadata object_namer4bucket_url_stringr r rpop_BuildBucketFilterStrings_GetRegexPatternsmatchr3_SingleVersionMatchesr append)rbucket_listing_fieldsexpand_top_level_bucketssingle_version_requestr1fieldbucket_listing_refrSurl obj_or_prefix get_objectrurls_needing_expansionr?r:prefix_wildcardsuffix_wildcardregex_patternslisting_fieldspattern gcs_objectrstripped_prefixurl_append_strings r__iter__zCloudWildcardIterator.__iter__ps0"..<<> J5j(%u)"33*4,""L*#=>! 4#4#4$$ !7 8 :<)9:; #999O,77   # # % $$%67##::oo,,((//* ; ,m %%)F)F)M)MM&&'8'4'9'9484E4E'GG&&'8-:L:LMM ,# " !!2!2!=!=>    & & (1B1B ::!!--!!--,,77**11! ;#J $$T%6%6%H%H%/373D3D4J3I%L L  "$5$($5$5$A$A$($5$5$@$@B* t/@/@/L/L!M""*#-$$%;%?%?%BC##==cooN 69o 11/B. /  -4I  $::oo!,,F0F((//# ; %m*''8+H+H+O+OO*// ==1(]:??-K0?.@122:3H3HI ..+$(,(9(9)?(> /AA &++#F+ )"3F<==$1#8 ==!12$)9_)L2C2BS2H2A3B)C%+112CD,,->GGS* %%uPN$%:;   sGG>TB(S8)A"T DTB!T0AT4T8T T T  Tc0|g}d|vr$|jdd}|j|n|}||fD](}|jds|j|dd*|Dcgc]*}tjt j |,c}Scc}w)aReturns list of regex patterns derived from the wildcard patterns. Args: wildcard_pattern (str): A wilcard_pattern to filter the resources. Returns: List of compiled regex patterns. This translates the wildcard_pattern and also creates some additional patterns so that we can treat ** in a/b/c/**/d.txt as zero or more folders. This means, a/b/c/d.txt will also be returned along with a/b/c/e/f/d.txt. z/**/r9**/N)replacerY startswithrecompilefnmatch translate)rwildcard_patternwildcard_patternsupdated_patternrgps rrVz'CloudWildcardIterator._GetRegexPatternss** !!)00=o/(o$o6   E "   -77H H6GBJJw((+ ,6G HH Hs!/Bc tj|}|s |}d}|}d}n|jdkDr'|d|j}||jd}nd}|}|jd}|dk7r|d|dz}|xsd|z}|j ds t |}||j d}|jd}|dk(rd}n||dzd}|jddk7r d}||z }d}nd}|jjd t|t|t|t|t|||||fS) aBuilds strings needed for querying a bucket and filtering results. This implements wildcard object name matching. Args: wildcard: The wildcard string to match to objects. Returns: (prefix, delimiter, prefix_wildcard, suffix_wildcard) where: prefix is the prefix to be sent in bucket GET request. delimiter is the delimiter to be sent in bucket GET request. prefix_wildcard is the wildcard to be used to filter bucket GET results. suffix_wildcard is wildcard to be appended to filtered bucket GET results for next wildcard expansion iteration. For example, given the wildcard gs://bucket/abc/d*e/f*.txt we would build prefix= abc/d, delimiter=/, prefix_wildcard=d*e, and suffix_wildcard=f*.txt. Using this prefix and delimiter for a bucket listing request will then produce a listing result set that can be filtered using this prefix_wildcard; and we'd use this suffix_wildcard to feed into the next call(s) to _BuildBucketFilterStrings(), for the next iteration of listing/filtering. Raises: AssertionError if wildcard doesn't contain any wildcard chars. r9rNrmz**zMwildcard=%s, prefix=%s, delimiter=%s, prefix_wildcard=%s, suffix_wildcard=%s ) rsearchstartfindendswithrendr.debugr) rwildcardrWr?r:rcrd wildcard_partrs rrUz/CloudWildcardIterator._BuildBucketFilterStrings9sw:  ! !( +E fi oo  .5;;=) 0    s #c %hsQw/ 26o  % %e ,(8 .o   %c )#'(3   d #r ) ?*  KK 34@4JVl95_%|O'D F I @@r ct|j|}t|jjt|k(SN)rrstrr4)rlisted_generationdecoded_generations rrXz+CloudWildcardIterator._SingleVersionMatchess;3D4E4E4EG t  ++ ,4F0G GGr c#Kt|jj}|rEt|tdgk(r-t |jj st |y|jjrst |jj sTt ||jj|jj |jj|ytj|jj }tj|}|j|}|r|j!d|jj#||j$|jjD]\}|j'|j(st|jjd|j(d}t ||^yw) aGExpands bucket and provider wildcards. Builds a list of bucket url strings that can be iterated on. Args: bucket_fields: If present, populate only these metadata fields for buckets. Example value: ['acl', 'defaultObjectAcl'] Yields: BucketListingRefereneces of type BUCKET. r6)r;r< root_objectzitems/id)r<r+r;z://r9N)rrrSrArrHrrFr* GetBucketrIrsrtrqrrrCrB ListBucketsr+rWr6)rr8 bucket_urlregexprogr<bucketr_s rrEz,CloudWildcardIterator._ExpandBucketWildcardss&d&7&7&I&IJJ#m,TF ; T..:: ; ++    $ $ &t00<<=  ,0OO,E,E$($5$5$A$A-1->->-E-E+8-F-: ;;  1 1 = =>e ZZ d$$]3f  :OO//__$$++0-& ::fii $&*&7&7&>&> &KL##CV< <-s F0G43AG4cV|r't}|D]}|jd|z|Sy)aPrepends 'items/' to the input fields and converts it to a set. This way field sets requested for GetBucket can be used in ListBucket calls. Note that the input set must contain only bucket or object fields; listing fields such as prefixes or nextPageToken should be added after calling this function. Args: get_fields: Iterable fields usable in GetBucket/GetObject calls. Returns: Set of fields usable in ListBuckets/ListObjects calls. zitems/N)rArB)rr1 list_fieldsr]s rrCz&CloudWildcardIterator._GetToListFieldss4Ek%5()  r c|r@|j4t|j|j}||jd|}n||j}t |}t ||S)ahCreates a BucketListingRef of type OBJECT from the arguments. Args: bucket_url_string: Wildcardless string describing the containing bucket. gcs_object: gsutil_api root Object for populating the BucketListingRef. with_version: If true, return a reference with a versioned string. Returns: BucketListingRef of type OBJECT. r>r)r4rrr3rr)rrSrhr=generation_str object_string object_urls rrMz#CloudWildcardIterator._GetObjectRefsh --91$2C2C2<2G2GIn#4joo#13m!2:??Cm%m4J zz BBr c:t||}t||S)aCreates a BucketListingRef of type PREFIX from the arguments. Args: bucket_url_string: Wildcardless string describing the containing bucket. prefix: gsutil_api Prefix for populating the BucketListingRef Returns: BucketListingRef of type PREFIX. r)rr )rrSr? prefix_urls rrOz#CloudWildcardIterator._GetPrefixRefs!&0A6&JKJ zv >>r c#BK|j|D]}|yw)ahIterates over the wildcard, returning refs for each expanded bucket. This ignores the object part of the URL entirely and expands only the the bucket portion. It will yield BucketListingRefs of type BUCKET only. Args: bucket_fields: Iterable fields to include in bucket listings. Ex. ['defaultObjectAcl', 'logging']. This function is responsible for converting these to listing-style format ['items/defaultObjectAcl', 'items/logging'], as well as adding any fields necessary for listing such as 'items/id'. API implemenation is responsible for adding pagination fields. If this is None, all fields are returned. Yields: BucketListingRef of type BUCKET, or empty iterator if no matches. r7N)rE)rr8blrs r IterBucketsz!CloudWildcardIterator.IterBucketss&&***G iHc#DK|j||D]}|yw)a}Iterates over the wildcard, yielding bucket, prefix or object refs. Args: bucket_listing_fields: If present, populate only these metadata fields for listed objects. expand_top_level_buckets: If true and the wildcard expands only to Bucket(s), yields the expansion of each bucket into a top-level listing of prefixes and objects in that bucket instead of a BucketListingRef to that bucket. Yields: BucketListingRef, or empty iterator if no matches. rZr[Nrk)rrZr[rs rIterAllzCloudWildcardIterator.IterAll s-}}3H6NP iPs c#fK|j|dD]}|js|yw)a&Iterates over the wildcard, yielding only object BucketListingRefs. Args: bucket_listing_fields: If present, populate only these metadata fields for listed objects. Yields: BucketListingRefs of type OBJECT or empty iterator if no matches. TrN)rkrP)rrZrs r IterObjectsz!CloudWildcardIterator.IterObjectss4}}3H6:<  t3d j5t7j8t:t=|zwxYww) a@Iterator that gets called when iterating over the file wildcard. In the case where no wildcard is present, returns a single matching file or directory. Args: bucket_listing_fields: Iterable fields to include in listings. Ex. ['size']. Currently only 'size' is supported. If present, will populate yielded BucketListingObject.root_object with the file name and size. Raises: WildcardException: if invalid wildcard found. Yields: BucketListingRef of type OBJECT (for files) or PREFIX (for directories) rbeforeNr{after*z5Invalid wildcard with more than 2 consecutive *s (%s)zSkipping symbolic link %s...r )rArrRFLAT_LIST_REGEXrWgrouprpWildcardExceptionlstriprsep_IterDirglobiglobrrrislinkr.infoisdirr rrUnicodeEncodeErrorr jointextwrapwrap_UNICODE_EXCEPTION_TEXTrepr) rrZ include_sizerrWbase_dirremaining_wildcard filepathsr expanded_url blr_objects rrkzFileWildcardIterator.__iter__^s$*9c"788  ,,H  ! !( +E X&s+h ;;w/  & &s +!*,4!56 6   -44RVV<--*<=i**X&i)(3l F   BGGNN8$< [[ KK  ;X F  77== "#L1 13?~h/T*#LjI I Ftyy MM1DNB C EF FFs-CG !AF4G 5A FG AG  G c#ZKtjj|d|k(r|dz }tjt j |dD]\}}}g}|D]}tjj ||}|j|s|j|nG|jsTtjj|st|jjd|||ddtj||D].} tjj |t| 0y#t$r\t!dj t#j$t&t)tjj || zwxYww)aAn iterator over the specified dir and wildcard. Args: directory (unicode): The path of the directory to iterate over. wildcard (str): The wildcard characters used for filename pattern matching. Yields: (str) A string containing the path to a file somewhere under the directory hierarchy of `directory`. Raises: ComandException: If this method encounters a file path that it cannot decode as UTF-8. r\T)topdownzSkipping symlink directory "%s"Nr)rr splitdrivewalksix ensure_textr _ExcludeDirrYr.rrrsfilterrUnicodeDecodeErrorr rrrr) r directoryrdirpathdirnames filenamesfiltered_dirnamesdirname full_dir_pathfs rrzFileWildcardIterator._IterDirsc  ww)$Q'944i)+ 0J8<)>$9' Wg6  .  " "7 +  ;;277>>-8 ++   W&@&CD D3))>." >6!mm3 gq!9:;<"=> >7 >s+B3F+6F+F+A%F((F+cb|jy|j\}}}|syt|jt|jd}|j |j j r|dd}|j|r)|jr|jjd|yy)zCheck a directory to see if it should be excluded from os.walk. Args: dir: String representing the directory to check. Returns: True if the directory should be excluded. NFr|z!Skipping excluded directory %s...T) rrrlenrprdelimrWr.r)rdirbase_url exclude_dirsexclude_pattern str_to_checks rrz FileWildcardIterator._ExcludeDirs ! 040B0B-X|_  '  ZH//013Lt00667!!"%l\*  h0!$$ Br rc"eZdZdZdZdZdZy)rz+Exception raised for invalid wildcard URLs.c<tj|||_yr) StandardErrorr/reason)rrs rr/zWildcardException.__init__:s4 DKr c d|jzSNzWildcardException: %srrs rrzWildcardException.__repr__> "T[[ 00r c d|jzSrrrs r__str__zWildcardException.__str__Arr N)r!r"r#r$r/rrr%r rrr7s311r rct|}|xstj}|jrt ||||St ||||S)aInstantiate a WildcardIterator for the given URL string. Args: url_str: URL string naming wildcard object(s) to iterate. gsutil_api: Cloud storage interface. Passed in for thread safety, also settable for testing/mocking. all_versions: If true, the iterator yields all versions of objects matching the wildcard. If false, yields just the live object version. project_id: Project id to use for bucket listings. exclude_tuple: (base_url, exclude_pattern), where base_url is top-level URL to list; exclude_pattern is a regex of paths to ignore during iteration. ignore_symlinks: For FileUrls, ignore symlinks during iteration if true. logger: logging.Logger used for outputting debug messages during iteration. If None, the root logger will be used. Returns: A WildcardIterator that handles the requested iteration. )rrr.)r)r+)rr,r- IsFileUrlrr')url_strr*r)r+rrr.r_s rCreateWildcardIteratorrEs\8 W%#  (W&&(&]]_ .;0?'- // !!+.:,6 88r )FNNFN)2r$ __future__rrrrrsrr,rrqrrgslib.bucket_listing_refrrr gslib.cloud_apir r r gslib.exceptionr gslib.storage_urlrrrrr"gslib.third_party.storage_apitoolsrrgslib.utils.constantsrgslib.utils.text_utilrrPY3 Exceptionrrrrrobjectrr'rrrrr%r rrs8&%'   8881$-,.82+,W&<.77-"**@A> Av A],]@B"uB+uBp 1 1 ).&*)-+0"& '8r