ydZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlZdZe j@dZ!e j@dZ"e#dde jHzdde jHzgZ%dZ&dZ'dddejPjRdddddejTjVejXjVdejZj\ddddfdZ/dZ0GddejbejdZ3Gdde3Z4Gdd e3Z5Gd!d"Z6d#Z7y)$z1Utilities for expanding wildcarded GCS pathnames.)absolute_import)division)unicode_literalsN)Iterator) api_factory) cloud_api)errors)request_config_factory) folder_util) storage_url)resource_reference)log) debug_outputz!Expected files but got stream: {}z\*{3,}z[*?\[\]].z..cd|jtjdjdS)Nr) rpartitionossep startswith)paths ;lib/googlecloudsdk/command_lib/storage/wildcard_iterator.py _is_hiddenr3s%   # . .s 33c>ttj|S)zChecks whether url_string contains a wildcard. Args: url_string: URL string to check. Returns: bool indicator. )boolWILDCARD_REGEXsearch) url_strings rcontains_wildcardr 7s n##J/ 00rTFctj|}t|tjrt ||||||||| | | | |||St|tj rt ||||| |Stjd|z)aInstantiate a WildcardIterator for the given URL string. Args: url_str (str): URL string which may contain wildcard characters. error_on_missing_key (bool): If true, and the encryption key needed to decrypt an object is missing, the iterator raises an error for that object. exclude_patterns (Patterns|None): Don't return resources whose URLs or local file paths matched these regex patterns. fetch_encrypted_object_hashes (bool): Fall back to GET requests for encrypted cloud objects in order to fetch their hash values. fields_scope (cloud_api.FieldsScope): Determines amount of metadata returned by API. files_only (bool): Skips containers. Raises error for stream types. Still returns symlinks. force_include_hidden_files (bool): Include local hidden files even if not recursive iteration. URL should be for directory or directory followed by wildcards. get_bucket_metadata (bool): If true, perform a bucket GET request when fetching bucket resources. halt_on_empty_response (bool): Stops querying after empty list response. See CloudApi for details. ignore_symlinks (bool): Skip over symlinks instead of following them. managed_folder_setting (folder_util.ManagedFolderSetting): Indicates how to deal with managed folders. folder_setting (folder_util.FolderSetting): Indicates how to deal with folders. next_page_token (str|None): Used to resume LIST calls. object_state (cloud_api.ObjectState): Versions of objects to query. preserve_symlinks (bool): Preserve symlinks instead of following them. raise_managed_folder_precondition_errors (bool): If True, raises precondition errors from managed folder listing. Otherwise, suppresses these errors. This is helpful in commands that list managed folders by default. soft_deleted_buckets (bool): If true, soft deleted buckets will be queried. list_filter (str|None): If provided, objects with matching filters will be returned, The prefixes would still be returned regardless of whether they match the specified filter, See go/gcs-object-context-filtering for more details. Returns: A WildcardIterator object. )error_on_missing_keyexclude_patternsfetch_encrypted_object_hashes fields_scope files_onlyget_bucket_metadatahalt_on_empty_responsemanaged_folder_settingfolder_settingnext_page_token object_state(raise_managed_folder_precondition_errorssoft_deleted_buckets list_filter)r#r&force_include_hidden_filesignore_symlinkspreserve_symlinkszUnknown url type %s.) r storage_url_from_string isinstanceCloudUrlCloudWildcardIteratorFileUrlFileWildcardIteratorcommand_errorsInvalidUrlError)url_strr"r#r$r%r&r0r'r(r1r)r*r+r,r2r-r.r/urls rget_wildcard_iteratorr=Cs~ ++G4#[))*  1)&C!/55%'!1Y1 "#{**+  )#='+    ( ()?#)E FFrctjtd|j}|j|d|jzz }t j |S)zAsterisk counts greater than two treated as single * to mimic globs. Args: url (StorageUrl): Url to compress wildcards in. Returns: StorageUrl built from string with compressed wildcards. *#)resubCOMPRESS_WILDCARDS_REGEXversionless_url_string generationr r3)r<compressed_url_strings r_compress_url_wildcardsrGsQ&&!93!$!;!;=^^S3>>11  , ,-B CCrc"eZdZdZ ddZdZy)WildcardIteratoraClass for iterating over Google Cloud Storage strings containing wildcards. The base class is abstract; you should instantiate using the wildcard_iterator() static factory method, which chooses the right implementation depending on the base string. Nc@t||_||_||_y)z@Initializes class. See get_wildcard_iterator for Args docstring.N)rG_url_exclude_patterns _files_only)selfr<r#r&s r__init__zWildcardIterator.__init__s (,DI-D!Drc6dt|jddzS)z2Returns string representation of WildcardIterator.zWildcardIterator(%s)rN)getattrrKrNs r__repr__zWildcardIterator.__repr__s !GDII|T$J JJr)NF)__name__ __module__ __qualname____doc__rOrSrrrIrIs "KrrIc4eZdZdZ dfd ZdZxZS)r8z,Class to iterate over files and directories.ctt| |||||_||_|rD|j j ddtjk7rtjd|jj |_ d|jv|_ |jxs|xst|j|_y)aeInitialize FileWildcardIterator instance. Args: url (FileUrl): A FileUrl instance representing a file path. exclude_patterns (Patterns|None): See get_wildcard_iterator. files_only (bool): Returns files and symlinks, skips folders, errors on streams. force_include_hidden_files (bool): Include hidden files even if not recursive iteration. URL should be for directory or directory followed by wildcards. ignore_symlinks (bool): Skip over symlinks instead of following them. preserve_symlinks (bool): Preserve symlinks instead of following them. )r&r?z`If force-including hidden files, input URL must be directory or directory followed by wildcards.**N)superr8rO_ignore_symlinks_preserve_symlinks resource_namerstriprrr9r:rK_path_recurser_include_hidden_files)rNr<r#r&r0r1r2 __class__s rrOzFileWildcardIterator.__init__s, . / ,D/D #    $ $S )" - 7  * * . ((DJDJJ&DM M3Mz$**7M rc# K|jjrj|jr|jj/drN|r |jFs@t&j(jQ|rt&j(jI|rtAjR|} |s8t&j(jI|rtj| |r%|jFrtjT| tj| |8yw) Nrr\r?c3:K|]}t|zywN)str).0p path_prefixs r z0FileWildcardIterator.__iter__..s"?A c!f?szSkipping symlink {}) is_symlink)+rKis_stdiorMr9r:_FILES_ONLY_ERROR_FORMATformatr`r FileObjectResourcerb_RELATIVE_PATH_SYMBOLSFileDirectoryResourcepathlibPath expanduserrootlistpartsrrjoinanchorendswithrrglobrLmatchrdrisfiler is_named_pipeislinkr_isdirr^rwarningexistsr7FileSymlinkPlaceholderResource) rN pathlib_path path_components_relative_to_rootpath_relative_to_rootrz current_working_directory_prefix path_iteratorrrpfile_urlrns @r__iter__zFileWildcardIterator.__iter__s yy  ,, $ + +DII,C,C D    1 1$)) <<  zz++  4 4TYY ?? << +668L*.l.@.@.D)E& ggll,LM  d d"jj%%d+ ggll+@#F(+RVV|$ zz=>4kkd#(()>?M   T%;%;%A%A$%G((Z-=  "''.."6  $ $T *..&--dii.E.EF    77>>$'j ))ww}}T"d&;&; )0067 **  d #$"9"977>>$'277==+>$$T*h BGGMM$/ 66x@@ $11 ??II 33   UsP P )NFFFF)rTrUrVrWrOr __classcell__res@rr8r8s#4 !&*XV rr8ceZdZdZdddej j dddejjejjdejjdddffd Z dZdZdZdd Zd Z dd Z dd Zd ZddZddZdZdZdZdedeej<fdZdedeej<fdZ dZ!xZ"S)r6zAClass to iterate over Cloud Storage strings containing wildcards.TNFctt| |||tj|j j |_||_||_ ||_ ||_ ||_ | |_ | |_| |_| |_| |_||_||_| t(j*j,ur6|j j. t(j*j0|_n| |_| t(j*j4u|_|j t(j*j0uxs&|j6xr|j j.du|_y)aInstantiates an iterator that matches the wildcard URL. Args: url (CloudUrl): CloudUrl that may contain wildcard that needs expansion. error_on_missing_key (bool): If true, and the encryption key needed to decrypt an object is missing, the iterator raises an error for that object. exclude_patterns (Patterns|None): See get_wildcard_iterator. fetch_encrypted_object_hashes (bool): Fall back to GET requests for encrypted objects in order to fetch their hash values. fields_scope (cloud_api.FieldsScope): Determines amount of metadata returned by API. files_only (bool): Returns cloud objects, not prefixes or buckets. Also skips directory placeholder objects, although they are technically objects. get_bucket_metadata (bool): If true, perform a bucket GET request when fetching bucket resources. Otherwise, bucket URLs without wildcards may be returned without verifying the buckets exist. halt_on_empty_response (bool): Stops querying after empty list response. See CloudApi for details. managed_folder_setting (folder_util.ManagedFolderSetting): Indicates how to deal with managed folders. folder_setting (folder_util.FolderSetting): Indicates how to deal with folders. next_page_token (str|None): Used to resume LIST calls. object_state (cloud_api.ObjectState): Versions of objects to query. raise_managed_folder_precondition_errors (bool): If True, raises precondition errors from managed folder listing. Otherwise, suppresses these errors. This is helpful in commands that list managed folders by default. soft_deleted_buckets (bool): If true, soft deleted buckets will be queried. list_filter (str|None): If provided, objects with matching contexts will be returned. The prefixes would still be returned regardless of whether they match the specified context, See go/gcs-object-context-filtering for more details. )r#r&N)r]r6rOrget_apirKscheme_client_error_on_missing_key_fetch_encrypted_object_hashes _fields_scope_get_bucket_metadata_halt_on_empty_response_managed_folder_setting_folder_setting_next_page_token _object_state)_raise_managed_folder_precondition_errors_soft_deleted_buckets _list_filterr ObjectStateLIVErELIVE_AND_NONCURRENT_object_state_for_listing SOFT_DELETED _soft_deleted _object_state_requires_expansion)rNr<r"r#r$r%r&r'r(r)r*r+r,r-r.r/res rrOzCloudWildcardIterator.__init__UsFn / .:0&&tyy'7'78DL!5D*GD'%D 3D#9D #9D )D+D%D0 2"6D#D  --222 II ,'0'<'<'P'Pd$'3d$%)>)>)K)KKD i33GGG A    ?499#7#74#? )rc#K|jr5|jjs|jjry|jjr;|jj |j |jD]}|y|jD]}|jjr|#|j|jj}|j|jj|D]/}|jr0|jj|jjr@|jr]t!|t"j$rB|jj&j)tj*r|j,dk(r|j.t0j2j4urt!|t"j6s|j8t0j:j4urt!|t"j<s,|2yw)Nr% soft_deleted) is_hns_bucketr)rMrK is_provider is_bucketr list_bucketsrr_fetch_buckets_is_hns_bucketr bucket_name_fetch_sub_bucket_resourcesrLrrDr4r ObjectResourcer`rCLOUD_URL_DELIMITERsizerr ManagedFolderSettingLIST_WITHOUT_OBJECTSManagedFolderResourcer FolderSettingFolderResource)rNbucket_resourcebucket_or_unknown_resourcerresources rrzCloudWildcardIterator.__iter__s TYY224 8K8K8M  yy!\\66))117/  )-(;(;(= $ 99   * * --(44@@ -::(44@@); h%%$*@*@*F*F$$;;+x);)J)JK((66??#77! *,,33HHI"0FF$$,,AAB"8-?-N-NONI )>sI Icx|jrtjj|jj vr{|j tjjk7rSt|tjr8|js+|js|jr\|jj|j |j"|j$j&|j |j(S|j*rt-j.|j0|j*|j2}t5|j6ddr]|jj|j |j"||j$j&|j |j(S|S)NrEr%r)decryption_key_hash_sha256r"decryption_key)rr Capability ENCRYPTIONr capabilitiesr FieldsScopeSHORTr4r r crc32c_hashmd5_hashkms_keyget_object_metadatabucketnamerKrErrr get_request_configr rrQ resource_args)rNrrequest_configs r_decrypt_resource_if_necessaryz4CloudWildcardIterator._decrypt_resource_if_necessarysW ++  + +t||/H/H H   )"7"7"="= = x!3!B!B C%%):):   ||// OO MMyy++++++ 0    , ,/BB  '/'J'J!%!;!;= >//1A4 H11oomm------ 2   Orcr |jj||jjt j |j|jj |j|j}|j|S#tj$rYytj$r}|jdk(rhdt|vr[|jjj!|jj"r"|jrt%j&dnYd}~yd}~wwxYw)z/Matches user input that doesn't need expansion.rizYou must specify a generationzGET failed with "must specify generation" error. This is expected for a soft-deleted object listed with a trailing slash. Falling back to a LIST call.N)rrrKr`r rrErrr api_errors NotFoundError GcsApiError status_coderkrr delimiterrdebug)rNrres r_try_getting_object_directlyz2CloudWildcardIterator._try_getting_object_directly s 11  )) ! ! 3 3DII >YY))))))2h 0 0 ::  # #  " !  ! ! --3 -Q7ii""++DII,?,?@    3    !sB B D6"D64A8D11D6c&t|jjxsG|jxs9|jjj |jj }|s|j|}|r|gS|j||S)z:Fetch all objects for the given bucket that match the URL.) r rKr`rrrrr_expand_object_path)rNrrneeds_further_expansiondirect_query_results rrz1CloudWildcardIterator._fetch_sub_bucket_resources/s $))112 >  0 0 > 99   ( ()<)< = # ==kJ #$$  # #K ??rc#K|jdu}|jtjjtjj fv} |r^t jj|jjvr.|r,|jj||jxsd}ng}|D]}|y#tj$r|jrYywxYww)Nrprefix)rrr rLIST_WITH_OBJECTSrrrMANAGED_FOLDERSrrlist_managed_foldersrrPreconditionFailedErrorr)rNrwildcard_partsis_recursive_expansionshould_list_managed_foldersmanaged_folder_iteratorrs r_get_managed_folder_iteratorz2CloudWildcardIterator._get_managed_folder_iterator>s,55="&">">((::((==C# %""22dll6O6OO$"&,,"C"C#N,A,A,IT#D# #%-(.  - - 7 7  8s+AC&A-C?C& C# C&"C##C&c#K|jdu}|jtjjuxr|}|jtjj fvxs|}|j r7|j jds|j dzn |j }nd}|rPtjj|jjvr |r|jj||}ng}|D]} | yw)N/r)rrr rLIST_AS_FOLDERSrrrrrFOLDERSrr list_folders) rNrrrris_list_as_foldersshould_list_foldersmodified_prefixfolder_iteratorrs r_get_folder_iteratorz*CloudWildcardIterator._get_folder_iteratorYs,55=  9 9 I II     % % : : < =    &&//4   # %$$ o   ( (DLL,E,E E " 11! 2o o# n$sD D c |jtjjur&|jtj jus |j r|jtjjuxr&|jtj ju}t|j }|s|sdnd}|jj||j |j|j||j|jxsd|j|j  }ng}|j#||}|j%|||} t'j(||| dS)NT) rrr%r(include_folders_as_prefixesr+rr,r/c.|jjSrj)r r)rs rz>CloudWildcardIterator._get_resource_iterator..sX11< >  , , A A     / /t||7P7P P o''j \\ , ,  *":":  # # os;C C#"C#c t|tjusp|jtj j tj jhvs0tjj|jjvr|S|jtj jur|s|S |j}|jj|j|j S#t"j$$r|cYSwxYw)z E ! $ , , .d *55dDII> ?@!!)00=o/(o$o6   E "   -77H H6GBJJw((+ ,6G HH Hs"/Cc#(K|j|}|D]x}|jjr.|jj|jjk7rG|D]-}|j |jj s)|xzyw)aJFilter out resources that do not match the wildcard_pattern. Args: resource_iterator (iterable): An iterable resource_reference.Resource objects. wildcard_pattern (str): The wildcard_pattern to filter the resources. Yields: resource_reference.Resource objects matching the wildcard_pattern N)r(rKrEr rr`)rNrr$regex_patternsr regex_patterns rrz'CloudWildcardIterator._filter_resourcests}--.>?N% ))      ) )TYY-A-A A)-   x33AA B. * &s BB Bct|jjr%|j|jjS|jj ry|j rm|j r%|j|jjS|jj|jj|jgStj|jgS)zFetch the bucket(s) corresponding to the url. Returns: An iterable of BucketResource or UnknownResource objects. )rr%) r rKr_list_buckets_matching_wildcardrrr-_fetch_all_soft_deleted_generations_of_bucketr get_bucketrr UnknownResourcerRs rrz$CloudWildcardIterator._fetch_bucketss ../  1 1$))2G2G HH    4#<#<  # #AA II ! !  ,, ! !))//-- " !00; <Initialize the CloudWildcardParts object. Args: prefix (str): The prefix string to be passed to the API request. This is the substring before the first occurrance of the wildcard. filter_pattern (str): The pattern to be used to filter out the results returned by the list_objects call. This is a substring starting from the first occurance of the wildcard upto the first delimiter. delimiter (str): The delimiter to be passed to the api request. suffix (str): The substirng after the first delimiter in the wildcard. N)rrrr)rNrrrrs rrOzCloudWildcardParts.__init__s!DK(DDNDKrcrt|\}}|j|\}}}d|vrd}|}d}|||||S)aCreate a CloudWildcardParts instance from a string. Args: string (str): String that needs to be splitted into different parts. delimiter (str): The delimiter to be used for splitting the string. Returns: WildcardParts object. r\N)_split_on_wildcard partition)clsstringrrwildcard_stringr_rs rrzCloudWildcardParts.from_stringsV18FO!0 9 9) DNAv ~i&nf v~y& 99rc,tj|Srj)r generic_reprrRs rrSzCloudWildcardParts.__repr__s  $ $T **rN) rTrUrVrWrO classmethodr r5CLOUD_URL_DELIMrrSrXrrrrs2O")4)=)=)M)M::6+rrcttj|}||dfS|j}|d|}||d}||fS)a3Split the string into two such that first part does not have any wildcard. Args: string (str): The string to be split. Returns: A 2-tuple where first part doesn't have any wildcard, and second part does have a wildcard. If wildcard is not found, the second part is empty. If string starts with a wildcard then first part is empty. For example: _split_on_wildcard('a/b/c/d*e/f/*.txt') => ('a/b/c/d', '*e/f/*.txt') _split_on_wildcard('*e/f/*.txt') => ('', '*e/f/*.txt') _split_on_wildcard('a/b/c/d') => ('a/b/c/d', '') Nrh)rrstart)rErfirst_wildcard_idxr wildcard_strs rrBrBsV    '% ] 2:{{} %% &&*+,,  r)8rW __future__rrrabcrr"rrrwrAtypingrgooglecloudsdk.api_lib.storagerrr rr "googlecloudsdk.command_lib.storager9r r ,googlecloudsdk.command_lib.storage.resourcesr googlecloudsdk.corergooglecloudsdk.core.utilrsixrrr!rCr frozensetrrurr rr=rrrrrr=rGwith_metaclassABCMetarIr8r6rrBrXrrr\sf 8&'   64?AG::K#1 ?%2::i0K("CrvvtTBFF]#KL4 1"'&&--$&;;GG,,88&&++-2%\G~ D K)s))#++6K0E +E PU ,U p1+1+hr