kdZddlmZddlmZddlmZddlmZddlmZddlmZddl m Z dd l m Z d Z d d iZd ddddddddddddd Zdddddd d!d"d#d$d%d&d'd( ZGd)d*eZGd+d,eZGd-d.eZGd/d0eZGd1d2eZGd3d4eZGd5d6eZGd7d8eZGd9d:eZGd;dZ dBd@Z dBdAZy?)CaGUtils for generating API-specific RequestConfig objects. RequestConfig is provider neutral and should be subclassed into a provider-specific class (e.g. GcsRequestConfig) by the factory method. RequestConfig can hold a BucketConfig or ObjectConfig. These classes also have provider-specific subclasses (e.g. S3ObjectConfig). )absolute_import)division)unicode_literals)encryption_util)errors) storage_url)log) debug_outputzapplication/octet-stream gzip_settingszGzip Transformsz(Setting Autoclass Terminal Storage ClasszSetting Default Object ACLzEnabling AutoclasszEnabling Hierarchical NamespacezSetting IP FilterzSetting Predefined Default ACLzPublic Access Preventionz Setting Recovery Point ObjectivezSetting Retention PeriodzLocking Retention PeriodzSetting Custom ContextszRemoving Custom ContextszUpdating Custom Contexts) autoclass_terminal_storage_classdefault_object_acl_fileenable_autoclassenable_hierarchical_namespaceip_filter_file_pathpredefined_default_object_aclpublic_access_preventionrecovery_point_objectiveretention_periodretention_period_to_be_lockedcustom_contexts_to_setcustom_contexts_to_removecustom_contexts_to_updatezSetting Custom TimezSetting Default Encryption Keyz Setting Default Event Based HoldzSetting Default Storage ClasszEnabling Object RetentionzSetting Event-Based Holdsz Setting Dual-Region for a BucketzPreserving ACLszSetting Time to Retain UntilzSetting Retention ModezSetting Soft Delete PolicieszSetting Temporary Holdsz#Setting Uniform Bucket Level Access) custom_timedefault_encryption_keydefault_event_based_holddefault_storage_classenable_per_object_retentionevent_based_hold placement preserve_acl retain_untilretention_modesoft_delete_durationtemporary_holduniform_bucket_level_accessc*eZdZdZ ddZdZdZy)_ResourceConfigaHolder for generic resource fields. Attributes: acl_file_path (None|str): Path to file with ACL settings. acl_grants_to_add (None|list[dict]): Contains API representations of ACL. For GCS, this looks like `{ 'entity': ENTITY, 'role': GRANT }`. acl_grants_to_remove: (None|list[str]): Identifier of entity to remove access for. Can be user, group, project, or keyword like "All". Nc.||_||_||_y)z-Initializes class, binding flag values to it.N) acl_file_pathacl_grants_to_addacl_grants_to_remove)selfr)r*r+s DM(D,D,D,D2D DO.D(D DO(D 4Dr/ct|t|stStt||xrD|j |j k(xr(|j|jk(xr |j|jk(xr|j|jk(xr|j|jk(xr|j|jk(xr|j|jk(xr|j|jk(xrj|j|jk(xrO|j|jk(xr4|j |j k(xr|j"|j"k(Sr1)r2r3r4rGrEr7rIrJrKrLrMrHrNrOrPrQrRrSr,r6rTs r-r7z_BucketConfig.__eq__sp eT$Z (  - -e 4 D   5#7#7 7 D  ! !U%;%; ; D  ! !U%;%; ; D  ! !U%;%; ; D  $ $(A(A A D MMU^^ + D OOu// / D  " "e&=&= = D   5#7#7 7 D OOu// / D   5#7#7 7 D  % %)C)C C Er/)NNNNNNNNNNNNNNNr?r@rArBr.r7 __classcell__rTs@r-rErEksH."!%$(" $ $ $#'!%""$(5>EEr/rEcpeZdZdZ dfd ZfdZxZS)_GcsBucketConfiga1Holder for GCS-specific bucket fields. See superclass for remaining attributes. Subclass Attributes: autoclass_terminal_storage_class (str|None): The storage class that objects in the bucket eventually transition to if they are not ' read for a certain length of time. default_encryption_key (str|None): A key used to encrypt objects added to the bucket. default_event_based_hold (bool|None): Determines if event-based holds will automatically be applied to new objects in bucket. default_object_acl_file_path (str|None): File path to default object ACL file. default_object_acl_grants_to_add (list[dict]|None): Add default object ACL grants to an entity for objects in the bucket. default_object_acl_grants_to_remove (list[str]|None): Remove default object ACL grants. default_storage_class (str|None): Storage class assigned to objects in the bucket by default. enable_autoclass (bool|None): Enable, disable, or don't do anything to the autoclass feature. Autoclass automatically changes object storage class based on usage. enable_per_object_retention (bool|None): Enable the object retention for the bucket. enable_hierarchical_namespace (bool|None): Enable heirarchical namespace during bucket creation. placement (list|None): Dual-region of bucket. public_access_prevention (bool|None): Blocks public access to bucket. See docs for specifics: https://cloud.google.com/storage/docs/public-access-prevention recovery_point_objective (str|None): Specifies the replication setting for dual-region and multi-region buckets. retention_period (int|None): Minimum retention period in seconds for objects in a bucket. Attempts to delete an object earlier will be denied. soft_delete_duration (int|None): Number of seconds objects are preserved and restorable after deletion in a bucket with soft delete enabled. uniform_bucket_level_access (bool|None): Determines if the IAM policies will apply to every object in bucket. c"Rtt| |||||||||||||| |!||_||_||_||_| |_| |_| |_ | |_ | |_ ||_ ||_ ||_||_||_||_||_||_||_||_yr1)rGr[r.r rrdefault_object_acl_file_path default_object_acl_grants_to_add#default_object_acl_grants_to_removerrrrrrrrrPrrr#r%)#r,r)r*r+r rIrrr]r^r_rrrrrrJrKrLrMrHrNrOrrrrPrrr#r%rQrRrSrTs# r-r.z_GcsBucketConfig.__init__sH D*"-MD)"8D$t|t|stStt||xr|j |j k(xr|j|jk(xr|j|jk(xr|j|jk(xr||j|jk(xr`|j|jk(xrD|j|jk(xr(|j|jk(xr |j|jk(xr|j|jk(xr|j |j k(xr|j"|j"k(xr|j$|j$k(xr|j&|j&k(xrj|j(|j(k(xrO|j*|j*k(xr4|j,|j,k(xr|j.|j.k(Sr1)r2r3r4rGr[r7r rrr^r_rrrrrrrrrPrrr#r%rVs r-r7z_GcsBucketConfig.__eq__(s! eT$Z (  ,U3 -  1 1  1 1 2 -  ' '5+G+G G -  ) )U-K-K K  -  1 1  1 1 2  -  4 4  4 4 5 -  & &%*E*E E -  ! !U%;%; ; -  , ,  , , - -  . .  . . / -  $ $(A(A A - NNeoo -! -"  ) )U-K-K K# -$  ) )U-K-K K% -&   5#7#7 7' -(  ! !U%;%; ;) -*  . .  . . /+ -.  % %)C)C C/ -0  , ,  , , -3r/)!NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNrWrYs@r-r[r[s'V'+!##''+*. "&$(##$("&EICVr/r[ceZdZdZy)_S3BucketConfigzHolder for S3-specific bucket fields. See superclass for attributes. We currently don't support any S3-only fields. This class exists to maintain the provider-specific subclass pattern used by the request config factory. Nr?r@rArBrCr/r-rbrbIsr/rbcVeZdZdZ dfd ZfdZxZS) _ObjectConfigaHolder for storage object settings shared between cloud providers. Superclass and provider-specific subclasses may add more attributes. Attributes: cache_control (str|None): Influences how backend caches requests and responses. content_disposition (str|None): Information on how content should be displayed. content_encoding (str|None): How content is encoded (e.g. "gzip"). content_language (str|None): Content's language (e.g. "en" = "English). content_type (str|None): Type of data contained in content (e.g. "text/html"). custom_contexts_to_set (dict|None): Custom contexts fields set by user. custom_contexts_to_remove (dict|None): Custom contexts fields to be removed by user. custom_contexts_to_update (dict|None): Custom contexts field to be added or changed by user. custom_fields_to_set (dict|None): Custom metadata fields set by user. custom_fields_to_remove (dict|None): Custom metadata fields to be removed. custom_fields_to_update (dict|None): Custom metadata fields to be added or changed. decryption_key (encryption_util.EncryptionKey): The key that should be used to decrypt information in GCS. encryption_key (encryption_util.EncryptionKey|None|CLEAR): The key that should be used to encrypt information in GCS or clear encryptions (the string user_request_args_factory.CLEAR). md5_hash (str|None): MD5 digest to use for validation. preserve_acl (bool): Whether or not to preserve existing ACLs on an object during a copy or other operation. size (int|None): Object size in bytes. storage_class (str|None): Storage class for cloud object. If None, will use bucket's default. ctt| |||||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||_ ||_||_||_||_||_||_yr1)rGrer. cache_controlcontent_dispositioncontent_encodingcontent_language content_typerrrcustom_fields_to_setcustom_fields_to_removecustom_fields_to_updatedecryption_keyencryption_keymd5_hashr size storage_class)r,r)r*r+rgrhrirjrkrrrrlrmrnrorprqr rrrsrTs r-r.z_ObjectConfig.__init__vs. -'(*>'D2D,D,D$D"8D%>D"%>D" 4D#:D #:D (D(DDM$DDI&Dr/ct|t|stStt||xr|j |j k(xr|j|jk(xr|j|jk(xr||j|jk(xr`|j|jk(xrD|j|jk(xr(|j|jk(xr |j|jk(xr|j|jk(xr|j|jk(xr|j |j k(xr|j"|j"k(xr|j$|j$k(xrj|j&|j&k(xrO|j(|j(k(xr4|j*|j*k(xr|j,|j,k(Sr1)r2r3r4rGrer7rgrhrirjrkrrrrlrmrnrorprqrrr rsrVs r-r7z_ObjectConfig.__eq__s eT$Z (  mT)%0 6   %"5"5 5 6  $ $(A(A A 6  ! !U%;%; ; 6  ! !U%;%; ;  6   !3!3 3  6  ' '5+G+G G  6  * *e.M.M M 6  * *e.M.M M 6  % %)C)C C 6  ( (E,I,I I 6  ( (E,I,I I 6   5#7#7 7 6   5#7#7 7 6 MMU^^ + 6 II # 6   !3!3 3! 6"   %"5"5 5%r/)NNNNNNNNNNNNNNNNNNNNrWrYs@r-rereRsW!J! $ $"" +*'Xr/rec\eZdZdZ dfd ZfdZxZS)_GcsObjectConfigagArguments object for requests with custom GCS parameters. See superclass for additional attributes. Attributes: event_based_hold (bool|None): An event-based hold should be placed on an object. custom_time (datetime|None): Custom time user can set. retain_until (datetime|None): Time to retain the object until. retention_mode (flags.RetentionMode|None|CLEAR): The key that should be used to set the retention mode policy in GCS or clear retention (the string user_request_args_factory.CLEAR). temporary_hold (bool|None): A temporary hold should be placed on an object. ctt| ||||||||| | | | | |||||||_||_||_||_||_y)N)r)r*r+rgrhrirjrkrrrrlrmrnrorprqrr)rGrvr.rrr!r"r$)r,r)r*r+rgrhrirjrkrrrrlrmrnrrorprrqr!r"rrr$rTs r-r.z_GcsObjectConfig.__init__sz4 D*#+1#/))!5";";1 7 7%% %+&#D,D$D(D(Dr/cnt|t|stStt||xr|j |j k(xrj|j|jk(xrO|j|jk(xr4|j|jk(xr|j|jk(Sr1) r2r3r4rGrvr7rrr!r"r$rVs r-r7z_GcsObjectConfig.__eq__s eT$Z (  "D 0 7 8    1 1 1 8  ! !U%;%; ; 8   !3!3 3 8   5#7#7 7  8   5#7#7 7 9r/)NNNNNNNNNNNNNNNNNNNNNNNrWrYs@r-rvrvs_ $! $ $"" 11)f99r/rvceZdZdZy)_S3ObjectConfigzBWe currently do not support any S3-specific object configurations.NrcrCr/r-rzrz sJr/rzc.eZdZdZ ddZdZdZy)_RequestConfiga<Holder for parameters shared between cloud providers. Provider-specific subclasses may add more attributes. Attributes: predefined_acl_string (str|None): ACL to set on resource. predefined_default_object_acl_string (str|None): Default ACL to set on resources. preserve_posix (bool|None): Whether to apply source POSIX metadata to destination. preserve_symlinks (bool|None): Whether symlinks should be preserved rather than followed. resource_args (_BucketConfig|_ObjectConfig|None): Holds settings for a cloud resource. NcJ||_||_||_||_||_yr1)predefined_acl_string$predefined_default_object_acl_stringpreserve_posixpreserve_symlinks resource_args)r,r~rrrrs r-r.z_RequestConfig.__init__s1"7D, -)D.D&Dr/cBt|t|stS|j|jk(xrj|j|jk(xrO|j |j k(xr4|j |j k(xr|j|jk(Sr1)r2r3r4r~rrrrr5s r-r7z_RequestConfig.__eq__/s eT$Z (  ""e&A&AA 6  5 5  5 5 6 6   5#7#7 7 6  " "e&=&= =  6   %"5"5 5 r/c,tj|Sr1r9r;s r-r<z_RequestConfig.__repr__;r=r/)NNNNNr>rCr/r-r|r|s'$!+/ ' +r/r|c>eZdZdZ dfd ZfdZxZS)_GcsRequestConfigaHolder for GCS-specific API request parameters. See superclass for additional attributes. Attributes: gzip_settings (user_request_args_factory.GzipSettings): Contains settings for gzipping uploaded files. no_clobber (bool): Do not copy if destination resource already exists. override_unlocked_retention (bool|None): Needed as confirmation for some changes to object retention policies. precondition_generation_match (int|None): Perform request only if generation of target object matches the given integer. Ignored for bucket requests. precondition_metageneration_match (int|None): Perform request only if metageneration of target object/bucket matches the given integer. c xtt| |||||_||_||_||_||_y)N)r~rr)rGrr.r no_clobberoverride_unlocked_retentionprecondition_generation_match!precondition_metageneration_match) r,r rrrrr~rrrTs r-r.z_GcsRequestConfig.__init__RsL T+3 0# ,% 'D DO'BD$)FD&-ND*r/cnt|t|stStt||xr|j |j k(xrj|j|jk(xrO|j|jk(xr4|j|jk(xr|j|jk(Sr1) r2r3r4rGrr7r rrrrrVs r-r7z_GcsRequestConfig.__eq__hs eT$Z (  -e4 3   %"5"5 5 3 OOu// / 3  , ,  , , - 3  . .  . . /  3  2 2  2 2 3 r/)NNNNNNNNrWrYs@r-rr@s2&"&$((, +/O,  r/rceZdZdZy)_S3RequestConfigzHolder for S3-specific API request parameters. Currently just meant for use with S3ObjectConfig and S3BucketConfig in the parent class "resource_args" field. NrcrCr/r-rrxsr/rcng}|D]$}t||d|j||&t|S)z@Takes user_args and unsupported_fields and returns feature list.N)getattrappendsorted) user_argsunsupported_fieldsresultfields r-,_extract_unsupported_features_from_user_argsrs> &!ey%&2 mm&u-." r/ct|dd}|rt|ddsd|_t|tt|tz}|r3t j djdj|t|t}|r4tjdjdj|yy)z=Raises error or logs warning if unsupported S3 field present.rNrzFeatures disallowed for S3: {}z, z(Some features do not have S3 support: {}) rrrS3_REQUEST_ERROR_FIELDSS3_RESOURCE_ERROR_FIELDSrErrorformatjoinS3_RESOURCE_WARNING_FIELDSr warning)user_request_argsuser_resource_argserror_fields_presentwarning_fields_presents r- _check_for_unsupported_s3_fieldsrs0/4H94!A7;423D3JL23E3KMM  ,,(// :N0OP H46KK:AA ()+,r/Nc t|tjsyt|dd}d} |j r|j tj vr|j tjjk(r@t} |rz|j| _ |j| _ |j| _ |j| _ |j| _|j| _|j | _|j"| _|j$| _|j&| _|j(| _|j*| _|j,| _|j.| _|j0| _|j2| _|j4| _|j6| _nG|j tjj8k(r t;} t=|n t?} t|dd| _ t|dd| _!t|dd| _"t|dd| _#t|dd| _$t|dd| _%t|d d| _&t|d d| _'t|d d| _(t|d d| _)t|d d| _*t|dd| _+no|jYr^|j tjjk(rbt[} |r|j\| _.|j^| _/|j`| _0|jb| _1|jd| _2nG|j tjj8k(rtg} t=|n ti} || _5|| _6|| _7|xstqjr| _:|rtqjv||r|nd| _<|r8|jj)|jjr|jj| _5n tz| _5|jl|jl| _6|j|| _>|j~| _?|j| _@|j| _A|j| _B|j| _C|j| _D|j| _E|j| _F|j| _G|j| _H|jr|jj| _I| r5|r3|j| _K|j| _L|j| _M| S)z@Generates metadata for API calls to storage buckets and objects.NrrHrIrJrKrLrMrNrOrPrQrRrS)Nr2rCloudUrlr is_bucketschemeVALID_CLOUD_SCHEMESProviderPrefixGCSr[r rrr]r^r_rrrrrrrrrrr#r%S3rbrrErHrIrJrKrLrMrNrOrPrQrRrS is_objectrvrrr!r"r$rzrerkrqrrrget_encryption_keyrpget_decryption_keyroDEFAULT_CONTENT_TYPErgrhrirjrlrmrnrrrr rsupperr)r*r+) urlrkdecryption_key_hash_sha256rperror_on_missing_keyrqrrrrnew_resource_argss r-!_get_request_config_resource_argsrsn C-- . 0/4H]]_ zz[444 {1155 5,.  AA  <!77  2!99  4!==  8!AA  <!DD  ?!66  1!11  ,!<<  7!>>  9!44  /);(D(D  % 99  4!99  4!11  ,!>>  9!55  0!<<  7 ::3366 6+-():;(/!();Z!N'.,d(4$)0.*6&)0.*6&)0.*6&,314-9)#*L$$0 *1/+7''./A/?(G$#*L$$0 '.,d(4$-42D.:* }} zz[//333*, (:(F(F%-?-P-P*);)H)H&+=+L+L(+=+L+L( {1144 4)+&'89(/%1"!)! >/<<>$!)8)K)K $-Act*M&  ( ( 4  * *+=+J+J  (+?  (  $ $ 0%7%@%@"(:(H(H%  0 0+,>+N+N(+=+N+N(  1 1,  4 4/  4 4/  3 3.  6 61  6 61(:'F'F$  ) )  , , 2 2 4 '-&8&F&F#*<*N*N'//* r/c t||||||||}|jtjjk(rt |} |r|j | _|j| _|jr|j| _|jrt|j| _ |jr[t|j| _ n@|jtjjk(r t|} n t|} t|dd| _t|dd| _t|dd| _t|dd| _|r|j(| _| Sd| _| S)zFGenerates API-specific RequestConfig. See output classes for arg info.)rr]Nr~rr)rrrrrrr rrrintrrrr|rr]r~rrr) rrkrrprrqrrrrrequest_configs r-get_request_configrLsw4 <3^Hd,=?- ZZ;--111&]CN%6%D%Dn"  7 70  % %$5$@$@!  8 87:  ; ;8=4  < <;>  ? ?rsl''>5:#12 & 3;,%F.%E : B2%?7!;!;$)> B<#>33%2.:/#H"+f+>EEOEEPS}SlmfOfRL9}L9^KmK.+V.+d55p~,248AE59;?/3+/8<hZ#  /r/