]&~dZddlmZddlmZddlmZddlZddlZddlZddlm Z ddl m Z ddl m Z dd lmZdd lmZdd lmZd ZGd dej(ZGddeZdZddZddZddZddZdZdZeZejAdZ!ejAdZ"ejAdZ#ddZ$y)aUtilities that augment the core CRC32C functionality for storage commands. The core CRC32C utility provides a hashlib-like functionality for CRC32C calculation but will at times fall back to a slow, all-Python implementation. This utility provides several mitigation strategies to avoid relying on the slow implementation of CRC32C, including adding a "deferred" strategy that uses the component gcloud-crc32c on files after they are downloaded. )absolute_import)division)unicode_literalsN) info_holder)errors)crc32c)binary_operations)log) propertiesz gcloud-crc32cc*eZdZdZfdZddZxZS)GcloudCrc32cOperationz1Operation for hashing a file using gcloud-crc32c.c 8tt| ddti|y)Nbinary)superr __init__ BINARY_NAME)selfkwargs __class__s :lib/googlecloudsdk/command_lib/storage/fast_crc32c_util.pyrzGcloudCrc32cOperation.__init__/s /M{MfMc 4dt|dt||gS)Nz-oz-l)str)r file_pathoffsetlengthrs r_ParseArgsForCommandz*GcloudCrc32cOperation._ParseArgsForCommand2s #f+tS[) <L)structpackr'r*s rdigestzDeferredCrc32c.digest`s ;;tTYY ''rcVdj|jjdS)zReturns a checksum like `digest` except as a bytestring of double length. Returns: A sixteen byte digest string, containing only hex digits. z{:08x}ascii)formatr'encoder*s r hexdigestzDeferredCrc32c.hexdigestjs" ??499 % , ,W 55rN)r) rr r!r"rr+r.r6r:r?rrrr%r%6s% ) ;&(6rr%ct ttjtvS#tj$rYywxYw)zCReturns if gcloud-crc32c is installed and does not attempt install.F)rr CheckForInstalledBinaryMissingExecutableExceptionrrr_is_gcloud_crc32c_installedrCss6 +CCKP PP  5 5 s !77c6 ttjt|vS#tj$rKt j dt jjjjdYyt j dYyxYw)zCReturns True if gcloud-crc32c is installed and optionally installs.install_if_missingzygcloud-crc32c can not be found on the path. The user may have opted out of installation or lack component manager access.Fz.Failed to check if gcloud-crc32c is available.) rr rArBr debugr VALUESstorageuse_gcloud_crc32cSetrEs r!_check_if_gcloud_crc32c_availablerL{s@ +CC(:   5 5 ; II A//33E: @II>? s #ABBctjxsAtjjj j duxr t|S)NF)rIS_FAST_GOOGLE_CRC32C_AVAILABLEr rHrIrJGetBoolrLrEs rcheck_if_will_use_fast_crc32crPsG  / / 1199;5H@ +,> ? rctjjjj }|dury|t j ry|r tdStS)zReturns True if gcloud-crc32c should be used and installs if needed. Args: install_if_missing (bool): Install gcloud-crc32c if not already present. Returns: True if the Go binary gcloud-crc32c should be used. FTrE) r rHrIrJrOrrNrLrC)rFuser_wants_gcloud_crc32cs rshould_use_gcloud_crc32crSs]1199;& & ,,  , EE $ &&rc\td}|r tStj|S)aZWraps the crc32c.get_crc32c() method to allow fallback to gcloud-crc32c. DO NOT USE for streaming downloads, as this relies on file-based hashing and does not take whether or not streaming is enabled into account. Args: initial_data (bytes): The CRC32C object will be initialized with the checksum of the data. Returns: A DeferredCrc32c instance if hashing can be deferred. Otherwise it returns a google_crc32c.Checksum instance if google-crc32c (https://github.com/googleapis/python-crc32c) is available and a predefined.Crc instance from crcmod library if not. TrE)rSr%r get_crc32c) initial_data should_defers rrUrUs) *TB,) Nv/@/@/NNrctj}|jj}|rGtj j |dd}dj|jj|Sy)zReturns the command to install google-crc32c library. This will typically only be called if gcloud-crc32c is missing and can't be installed for some reason. It requires user intervention which is why it's not a preferred option. lib third_partyz5{} -m pip install google-crc32c --upgrade --target {}N) r InfoHolder installationsdk_rootospathjoinr=basicpython_location)sdk_infor]third_party_paths r!get_google_crc32c_install_commandresc # # %(  " " + +( ww||He]C B I I&&(8 :: rc~t}d}tjdj|r |S|S)z CRC32C warnings share this text.z'gcloud components install gcloud-crc32cz This copy {{}} since fast hash calculation tools are not installed. You can change this by running: $ {crc32c_step} You can also modify the "storage/check_hashes" config setting.) crc32c_step)retextwrapdedentr=)google_crc32c_install_stepgcloud_crc32c_install_steps r_get_hash_check_warning_baserlse ABH HIO '1IOI  *IOI  rzwill not be validatedz may be slowz was skippedc tdrytjjjj }|tj jjk(r|rtjty|tj jjk(rtjty|tj jjk(rtj t"y)aInforms user about slow hashing if requested. Args: warn_for_always (bool): User may not want to see a warning about slow hashes if they have the "always check hashes" property set because (1) they intentionally set a property and (2) it could duplicate a warning in FileDownloadTask. Raises: errors.Error: IF_FAST_ELSE_FAIL set, and CRC32C binary not present. See error message for more details. TrEN)rPr rHrI check_hashesGet CheckHashesALWAYSvaluer warning_SLOW_HASH_CHECK_WARNINGIF_FAST_ELSE_SKIP_NO_HASH_CHECK_WARNINGIF_FAST_ELSE_FAILrr0_NO_HASH_CHECK_ERROR)warn_for_alwaysrns rlog_or_raise_crc32c_issuesrzs#d; ""**77;;=,Z++22888_KK()z--??EEEKK&'z--??EEE ,,+ ,,Fr)F)r)T)%r" __future__rrrr^r8rhgooglecloudsdk.command_libr"googlecloudsdk.command_lib.storagergooglecloudsdk.command_lib.utilr&googlecloudsdk.command_lib.util.anthosr googlecloudsdk.corer r rBinaryBackedOperationr objectr%rCrLrPrSrUrerl_HASH_CHECK_WARNING_BASEr=rvrtrxrzrrrrs'' 252D#* =-CC=:6V:6z,'0O(  $891884::=I/66}E-r