wdZddlmZddlmZddlmZddlmZddlZddlZddlZddl Z ddl Z ddl m Z ddl Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlZ ddlmZGddeZGddeZGddeZ d&dZGddeZ GddeZ!GddeZ"GddeZ#ejHddd gZ%ejHd!d"d#gZ&Gd$d%e jNZ(y)'aBName expansion iterator and result classes. Name expansion support for the various ways gsutil lets users refer to collections of data (via explicit wildcarding as well as directory, bucket, and bucket subdir implicit wildcarding). This class encapsulates the various rules for determining how these expansions are done. )absolute_import)print_function)division)unicode_literalsN)encoding)CommandException)NO_URLS_MATCHED_GENERIC)NO_URLS_MATCHED_TARGET)PluralityCheckableIterator)SeekAheadResult)storage_v1_messages)StorageUrlFromStringceZdZdZdZdZy)NameExpansionResulta*Holds one fully expanded result from iterating over NameExpansionIterator. The member data in this class need to be pickleable because NameExpansionResult instances are passed through Multiprocessing.Queue. In particular, don't include any boto state like StorageUri, since that pulls in a big tree of objects, some of which aren't pickleable (and even if they were, pickling/unpickling such a large object tree would result in significant overhead). The state held in this object is needed for handling the various naming cases (e.g., copying from a single source URL to a directory generates different dest URL names than copying multiple URLs to a directory, to be consistent with naming rules used by the Unix cp command). For more details see comments in _NameExpansionIterator. c||_||_||_||_||_|rt j ||_yd|_y)aInstantiates a result from name expansion. Args: source_storage_url: StorageUrl that was being expanded. is_multi_source_request: bool indicator whether multiple input URLs or src_url_str expanded to more than one BucketListingRef. is_multi_top_level_source_request: same as is_multi_source_request but measured before recursion. names_container: Bool indicator whether src_url names a container. expanded_storage_url: StorageUrl that was expanded. expanded_result: cloud object metadata in MessageToJson form (for pickleability), if any was iterated; None otherwise. Consumers must call JsonToMessage to get an apitools Object. N)source_storage_urlis_multi_source_request!is_multi_top_level_source_requestnames_containerexpanded_storage_urlr MessageToJsonexpanded_result)selfrrrrrrs 'platform/gsutil/gslib/name_expansion.py__init__zNameExpansionResult.__init__@sR"1D#:D -ND**D 4D+$11D15 c d|jzS)Nz%s)rrs r__repr__zNameExpansionResult.__repr__Ys $++ ++rN)__name__ __module__ __qualname____doc__rrrrrr/s 62,rrc0eZdZdZ ddZdZdZy)_NameExpansionIteratorzlClass that iterates over all source URLs passed to the iterator. See details in __iter__ function doc. Nc .||_||_||_||_||_||_||_|jj|j_||_ | |_ | |_ | |_ | s tdgn| |_ddd|_y)a Creates a NameExpansionIterator. Args: command_name: name of command being run. debug: Debug level to pass to underlying iterators (range 0..3). logger: logging.Logger object. gsutil_api: Cloud storage interface. Settable for testing/mocking. url_strs: PluralityCheckableIterator of URL strings needing expansion. recursion_requested: True if -r specified on command-line. If so, listings will be flattened so mapped-to results contain objects spanning subdirectories. all_versions: Bool indicating whether to iterate over all object versions. cmd_supports_recursion: Bool indicating whether this command supports a '-r' flag. Useful for printing helpful error messages. project_id: Project id to use for bucket retrieval. ignore_symlinks: If True, ignore symlinks during iteration. continue_on_error: If true, yield no-match exceptions encountered during iteration instead of raising them. bucket_listing_fields: Iterable fields to include in expanded results. Ex. ['name', 'acl']. Underyling iterator is responsible for converting these to list-style format ['items/name', 'items/acl']. If this is None, only the object name is included in the result. Examples of _NameExpansionIterator with recursion_requested=True: - Calling with one of the url_strs being 'gs://bucket' will enumerate all top-level objects, as will 'gs://bucket/' and 'gs://bucket/*'. - 'gs://bucket/**' will enumerate all objects in the bucket. - 'gs://bucket/abc' will enumerate either the single object abc or, if abc is a subdirectory, all objects under abc and any of its subdirectories. - 'gs://bucket/abc/**' will enumerate all objects under abc or any of its subdirectories. - 'file:///tmp' will enumerate all files under /tmp, as will 'file:///tmp/*' - 'file:///tmp/**' will enumerate all files under /tmp or any of its subdirectories. Example if recursion_requested=False: calling with gs://bucket/abc/* lists matching objects or subdirs, but not sub-subdirs or objects beneath subdirs. Note: In step-by-step comments below we give examples assuming there's a gs://bucket with object paths: abcd/o1.txt abcd/o2.txt xyz/o1.txt xyz/o2.txt and a directory file://dir with file paths: dir/a.txt dir/b.txt dir/c/ namez***)TFN) command_namedebuglogger gsutil_apiurl_strsrecursion_requested all_versions HasPlurality has_pluralitycmd_supports_recursion project_idignore_symlinkscontinue_on_errorsetbucket_listing_fields_flatness_wildcard) rr*r+r,r-r.r/r0r3r4r5r6r8s rrz_NameExpansionIterator.__init__csB%DDJDK DODM2D$D#'--"<"<">DMM"8D DO*D.D7L#vh-'<  &*#6Drc #zK|jD]u}t|}|jrU|js|j r5|jj r t dt|ddd|dtd}|jrH|jr8|js,t|j|jdg}nWt|j|j|jd}|jr|jrd}|j!}|jj xs|}|j"|j}|jrt%||||j}n t'|}t|}|j)r1|j*r t t,|zt t,|ztt3||j|j4|j6|j8} | j!}|jj xs|} | D]"\} } |xs| }| j;r(t|| ||| j<| j>Et| j@}|jr| tBjD|}n|jG| }t|j|jI|j }|xs|j!}|jj xs|} |D])} t|| |d| j<| j>+%xy#t $r&} | t/j0d fYd} ~ d} ~ wwxYww) a9Iterates over all source URLs passed to the iterator. For each src url, expands wildcards, object-less bucket names, subdir bucket names, and directory names, and generates a flat listing of all the matching objects/files. You should instantiate this object using the static factory function NameExpansionIterator, because consumers of this iterator need the PluralityCheckableIterator wrapper built by that function. Yields: gslib.name_expansion.NameExpansionResult. Raises: CommandException: if errors encountered. zPMultiple URL strings are not supported with streaming ("-") URLs or named pipes.FN)rrrrrrid) bucket_fieldsT)r8expand_top_level_bucketswildcard_suffixr8)%r.r IsFileUrlIsStreamIsFifor2rr IsCloudUrlIsBucketr/r WildcardIterator IterBucketsIterAllr8r1r9_ImplicitBucketSubdirIterator_NonContainerTuplifyIteratorIsEmptyr6r sysexc_info_OmitNonRecursiveIteratorr*r3r,IsObject storage_url root_object url_stringossepCreatePrefixUrl IterObjects)rurl_strrQsrc_names_bucketpost_step1_itersrc_url_expands_to_multirsubdir_exp_wildcardpost_step2_iterepost_step3_iterrrblrsrc_names_container expanded_urlurl_to_iteratewc_iters r__iter__z_NameExpansionIterator.__iter__s"==(1k    !    ![%7%7%9 == & & "MN N![:?DI277B26 8 8    "{';';'=&& 5  ! !' * 6 6dV 6 LN 5  ! !' * 2 2&*&@&@)- 3 /0  ! ! # (<(<(>! !0!=!=!?+/==+F+F,D+C(!33D4L4LM ! !7 /#6  & &(7G2?Co  "  ! ! )"#9G#CD D !!7'!AB B3 #OT5M5M$($5$5$($?$? NOo "1!=!=!?!%!#!,&=/1#&??!oo/ /.cnn=,  # # %),bff6IJN*99 3:5N/##N3??(,(B(B@DE''?'=&-&:&:&< #%)]]%@%@&>%= "c%#.(?5 $%(__ #1 1I%4s!D" )cllnQ'( ( ( )s1G N;N !F(N; N8N3-N;3N88N;ctjj||j|j|j |j |jS)a2Helper to instantiate gslib.WildcardIterator. Args are same as gslib.WildcardIterator interface, but this method fills in most of the values from instance state. Args: url_string: URL string naming wildcard objects to iterate. Returns: Wildcard iterator over URL string. )r0r4r5r,)gslibwildcard_iteratorCreateWildcardIteratorr-r0r4r5r,)rrSs rrGz'_NameExpansionIterator.WildcardIteratorQsL  " " 9 9 &&??,,{{ : rFTNFFN)r r!r"r#rrerGr$rrr&r&]s."&*$!&%)T7lV1prr&c(eZdZdZ ddZdZy)SeekAheadNameExpansionIteratorzCreates and wraps a _NameExpansionIterator and yields SeekAheadResults. Unlike the NameExpansionIterator, which can make API calls upon __init__ to check for plurality, this iterator does no work until the first iteration occurs. Nc |dvxr| |_|jrdgnd} t||tjd|t |||||| d|  |_y)z5Initializes a _NameExpansionIterator with the inputs.)cpmvrewritesizeNdummyTr0r3r4r5r6r8)count_data_bytesr&logging getLoggerr name_expansion_iterator) rr*r+r-r.r/r0r3r4r5file_size_will_changer8s rrz'SeekAheadNameExpansionIterator.__init__nsu *-DD7!66  )-(=(=VH4#9 '""8,!5'3 $5D rc#K|jD]s}|jrY|jrMtjt j |j}|jxsd}t|htuyw)Nr) data_bytes) rwrtrr JsonToMessageapitools_messagesObjectrqr )rname_expansion_resultiterated_metadata iterated_sizes rrez'SeekAheadNameExpansionIterator.__iter__sq!%!=!=  #8#H#H$22  $ $&;&K&KM)..3! 77">sBB)FTNFFr r!r"r#rrer$rrrlrlfs#"&*$%*#5J rrlc t|}t|||||||||| | |  } t| } | jrtt| S)arStatic factory function for instantiating _NameExpansionIterator. This wraps the resulting iterator in a PluralityCheckableIterator and checks that it is non-empty. Also, allows url_strs to be either an array or an iterator. Args: command_name: name of command being run. debug: Debug level to pass to underlying iterators (range 0..3). logger: logging.Logger object. gsutil_api: Cloud storage interface. Settable for testing/mocking. url_strs: Iterable URL strings needing expansion. recursion_requested: True if -r specified on command-line. If so, listings will be flattened so mapped-to results contain objects spanning subdirectories. all_versions: Bool indicating whether to iterate over all object versions. cmd_supports_recursion: Bool indicating whether this command supports a '-r' flag. Useful for printing helpful error messages. project_id: Project id to use for the current command. ignore_symlinks: If True, ignore symlinks during iteration. continue_on_error: If true, yield no-match exceptions encountered during iteration instead of raising them. bucket_listing_fields: Iterable fields to include in expanded results. Ex. ['name', 'acl']. Underyling iterator is responsible for converting these to list-style format ['items/name', 'items/acl']. If this is None, only the object name is included in the result. Raises: CommandException if underlying iterator is empty. Returns: Name expansion iterator instance. For example semantics, see comments in NameExpansionIterator.__init__. rs)r r&rLrr ) r*r+r,r-r.r/r0r3r4r5r6r8rws rNameExpansionIteratorrsj^( 1(2  3%)1 377NO$$& 2 33  rceZdZdZdZdZy)rKzIterator that produces the tuple (False, blr) for each iterated value. Used for cases where blr_iter iterates over a set of BucketListingRefs known not to name containers. c||_y)zTInstantiates iterator. Args: blr_iter: iterator of BucketListingRef. Nblr_iter)rrs rrz%_NonContainerTuplifyIterator.__init__s DMrc#:K|jD]}d|f yw)NFr)rr`s rrez%_NonContainerTuplifyIterator.__iter__s}} CLsNrr$rrrKrKs rrKceZdZdZdZdZy)rOaIterator wrapper for that omits certain values for non-recursive requests. This iterates over tuples of (names_container, BucketListingReference) and omits directories, prefixes, and buckets from non-recurisve requests so that we can properly calculate whether the source URL expands to multiple URLs. For example, if we have a bucket containing two objects: bucket/foo and bucket/foo/bar and we do a non-recursive iteration, only bucket/foo will be yielded. cJ||_||_||_||_||_y)aInstanties the iterator. Args: tuple_iter: Iterator over names_container, BucketListingReference from step 2 in the NameExpansionIterator recursion_requested: If false, omit buckets, dirs, and subdirs command_name: Command name for user messages cmd_supports_recursion: Command recursion support for user messages logger: Log object for user messages N) tuple_iterr/r*r3r,)rrr/r*r3r,s rrz"_OmitNonRecursiveIterator.__init__s*!DO2D$D"8DDKrc#K|jD]\}}|js|jst|j}|j rd}n |j }|jr3|jjd||j|j|jjd||j||fyw)N directoryz-Omitting %s "%s". (Did you mean to do %s -r?)zOmitting %s "%s".) rr/rPrrSrB type_namer3r,infor*)rrr`rbdescs rrez"_OmitNonRecursiveIterator.__iter__s"&//#  % %clln,CNN;  ! ! #$$  & & ++  J1B1B D ++  .cnn E$$#2sCCNrr$rrrOrOs $%rrOceZdZdZdZdZy)rJaIterator wrapper that performs implicit bucket subdir expansion. Each iteration yields tuple (names_container, expanded BucketListingRefs) where names_container is true if URL names a directory, bucket, or bucket subdir. For example, iterating over [BucketListingRef("gs://abc")] would expand to: [BucketListingRef("gs://abc/o1"), BucketListingRef("gs://abc/o2")] if those subdir objects exist, and [BucketListingRef("gs://abc") otherwise. c<||_||_||_||_y)aInstantiates the iterator. Args: name_exp_instance: calling instance of NameExpansion class. blr_iter: iterator over BucketListingRef prefixes and objects. subdir_exp_wildcard: wildcard for expanding subdirectories; expected values are ** if the mapped-to results should contain objects spanning subdirectories, or * if only one level should be listed. bucket_listing_fields: Fields requested in enumerated results. N)rname_exp_instancer\r8)rrrr\r8s rrz&_ImplicitBucketSubdirIterator.__init__3s#DM.D2D!6Drc#K|jD]}|jrt|jj |j }t |jj|j|j}|js|D]}d|f d|f|jrd|ftd|zyw)Nr?rATFz7_ImplicitBucketSubdirIterator got a bucket reference %s)rIsPrefixrrSrVr\r rrGrIr8rLrPr)rr` prefix_urlimplicit_subdir_iteratorexp_blrs rrez&_ImplicitBucketSubdirIterator.__iter__Es}} )#..9II 44J6 #=  " " 3 3J ? G G&*&@&@ H B$C (//11g/ !2   <<>cl E KM M%sCCNrr$rrrJrJ's 7$MrrJceZdZdZdZy)CopyObjectInfozARepresents the information needed for copying a single object. c|j|_|j|_|j|_|j|_|j|_|j |_||_||_y)aEInstantiates the object info from name expansion result and destination. Args: name_expansion_result: StorageUrl that was being expanded. exp_dst_url: StorageUrl of the destination. have_existing_dst_container: Whether exp_url names an existing directory, bucket, or bucket subdirectory. N)rrrrrr exp_dst_urlhave_existing_dst_container)rr~rrs rrzCopyObjectInfo.__init__`sk4FFD#8#P#PD ?? *0@@D 5 J JD0@@D"D'BD$rN)r r!r"r#rr$rrrr\s CrrDestinationInforr%NameExpansionIteratorDestinationTuplename_expansion_iter destinationc"eZdZdZdZdZdZy)CopyObjectsIteratoraIterator wrapper for copying objects and keeping track of source URL types. This is used in the cp command for copying from multiple source to multiple destinations. It takes a list of NameExpansionIteratorDestinationTuple. It wraps them and return CopyObjectInfo objects that wraps NameExpansionResult with the destination. It's used also for collecting analytics PerformanceSummary info, because there may be multiple source URLs and we want to know if any of them are file URLs, if any of them are cloud URLs, if any of them require daisy chain operations, and if any use different providers. The source URL type information will be aggregated at the end of _SequentialApply or _ParallelApply. c||_d|_d|_g|_||_t |j}|j |_|j|_ y)aInstantiates the iterator. Args: name_expansion_dest_iter: NameExpansionIteratorDestinationTuple iterator. is_daisy_chain: The -D option in cp might have already been specified, in which case we do not need to check again for daisy chain operations. FN) is_daisy_chain has_file_src has_cloud_srcprovider_typesname_expansion_dest_iternextrcurrent_expansion_iterrcurrent_destination)rrrname_expansion_dest_tuples rrzCopyObjectsIterator.__init__sY)DDDD$#>? .11==11MM OC   !7!7!A!A!Cd   #"8"8"C"C"Ed ++668//;;BBnn   N$> ))+ %%7 d $$D,?,??   !7!7!>!>? JA "&t'D'D"E # 7 7 !!:!F!Fd ]]_ sFAGGN)r r!r"r#rrerr$rrrrs E$$rrrj))r# __future__rrrr collectionsrurTrMsixapitools.base.pyrrggslib.exceptionrr r "gslib.plurality_checkable_iteratorr gslib.seek_ahead_threadr "gslib.third_party.storage_apitoolsr r|gslib.wildcard_iteratorrobjectrr&rlrrKrOrJr namedtuplerrIteratorrr$rrrs'%' % ,32I3W8+,&+,\FVFR5 V5 |(-15%)*/,104@!F6(/%/%d2MF2MjCVC6)+((  & )? (>(>+.)%G#,,Gr