BddlmZmZddlmZmZmZmZ ddlm Z ddl Z ddl mZmZmZmZddlmZmZmZddlmZdd lmZmZdd lmZmZmZmZm Z m!Z!e jDd Z#e#jIe jJe jLZ'e'jQe jRd e#jUe' dd e+de,de,de-dee dee de.de.defdZ/ ddede,de,de-dee dee de.de.defdZ0 dde de,de,de-dee dee de.de.defdZ1 dde de,de,de-dee dee de.defdZ2y#e $re Z Y@wxYw))basenamesplitext)BinaryIOListOptionalSet)PathLikeN)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encoding iana_nameidentify_sig_or_bom is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainreturnc t|ttfs#tdj t ||s$t jtjn#t jtjt|}|dk(r/t jdtt|dddgdgS|?t jd d j||D cgc]} t!| d}} ng}|?t jd d j||D cgc]} t!| d}} ng}|||zkrt jd |||d }|}|d kDr||z |krt#||z }t|t$k} t|t&k\} | r%t jdj |n&| r$t j)dj |g} |dur t+|nd} | '| j-| t j)d| t/}g}g}d}d}d}t}t1|\}}|1| j-|t j)dt||| j-dd| vr| j-d| t2zD]+}|r||vr |r||vr||vr|j5|d}||k(}|xr t7|}|dvr|durt j)d|] t9|} | r9|dur5tA|dur|dt#dn|t|t#d|ntA|dur|n |t|d|}d}|D]}tG||sd}n|rt jd|tI|durdn t||t#||z }|xr|duxrt||k} | rt j)d|t#t|dz }!|!dkrd}!d}"g}#g}$|D]}%||%|%|z}&|r |dur||&z}&|&jK|d}'|rc|%dkDr^||%d k\rV|d!kDrd!n|}(|rK|'d|(|vrDtI|%|%dz d"D]1})||)|%|z}&|r |dur||&z}&|&jK|d}'|'d|(|vs1n|#j-|'|$j-tM|'||$d"|k\r|"d z }"|"|!k\s|s|dusn|$rtO|$t|$z }*nd}*|*|k\s|"|!k\rb|j-|t jd#||"tQ|*d$zd%&|dd| fvr"t|||dg|}+|| k(r|+}n |dk(r|+}n|+}t j)d'|tQ|*d$zd%&|s tS|},n tU|},|,r.t j)d(j |tA|,g}-|#D]3}'tW|'d)|,rd*j|,nd}.|-j-|.5tY|-}/|/r%t j)d+j |/||j-t|||*||/||| ddfvr,|*d)kr't j)d,|t||gcS||k(st j)d-|t||gcSt|dk(r|s|s|rt jd.|r3t jd/|jZ|j-||S|r||r|r|j\|j\k7s|(t jd0|j-||S|r&t jd1|j-||Scc} wcc} w#t:t<f$rt j?d|Y4wxYw#tBtDf$rL}t|tDs t jd|tA||j-|Yd}~d}~wwxYw)2aD Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. z4Expected object of type bytes or bytearray, got: {0}rzXGiven content is empty, stopping the process very early, returning empty utf_8 str matchutf_8gFNz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).Tz@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %szW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.ignore)errorszc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g?,z We detected language {} using {}z0%s is most likely the one. Stopping the process.z[%s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z#%s will be used as a fallback matchz&utf_8 will be used as a fallback matchz&ascii will be used as a fallback match)/ isinstance bytearraybytes TypeErrorformattypeloggersetLevelloggingCRITICALINFOlenwarningrrjoinrintrrinforappendsetrraddrrModuleNotFoundError ImportErrordebugstrUnicodeDecodeError LookupErrorrrangedecodersumroundr r r rr+ fingerprint)0rrrrr r!r"r#lengthcpis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failurefallback_ascii fallback_u8fallback_specifiedresults sig_encoding sig_payload encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoderesimilar_soft_failure_testencoding_soft_failedr_multi_byte_bonusmax_chunk_gave_upearly_stop_count md_chunks md_ratiosi cut_sequencechunkchunk_partial_size_chkjmean_mess_ratiofallback_entrytarget_languages cd_ratioschunk_languagescd_ratios_mergeds0 1platform/bq/third_party/charset_normalizer/api.py from_bytesr~&sS 2 i)U!3 4 B I IY   (() % ^F { f |IwUBPRSTUU 5 IIl # 8DD| "e,| D  6 IIl # 8DD| "e,| D  *u$% l       qyVe^j0%(  N-?? N.>> L S S   W ^ ^  .BT-Iy)t%$$%78 N  UF  NKG 3I >L+$$\2 W      )++$$W-.? M=  M\9  F "  =!+}</ 4K 5  0 05IU5R KKm   $:=$I ! $)>%)G'50kD *"3{#3c$i@* #&'50"3{#3#56* # %*!$; ],@A,0)%< % NNi$   %.AC 4D     " .t+ .O$v-   KK-   B! , q !   A$QZ8L#(8E(A*\9  '' h'GE%Q9Q<43G%r/Bz' $556oM"1a!eR0'0Q^'D /4D4M+6+EL , 3 3M( 3 S !8"89_L!1   U #   Zy9 :} ) A%  $55$)9U)BSV !)ns9~=O!O i '+;?P+P # * *= 9 NN0 o+Q7  '3E FF!-}iO"!$66)7&"g-%3N"0K  K  /C' 3 %1-@ 4]C   KK8??!3'7#8   E-s:JCHH%56PTO   _ - 2)<  KK299$m   $     0'7C C#% KKBM "7=#9":; ; L ( KKm "7=#9":; ;U@X 7|q .,> NNa   NN57I7R7R  NN- . N^3"++~/I/II' NNC D NN; ' N  NNC D NN> * No EEF$[1  LLDm    *#K0 a-O!F $ * *= 9  s8-_0_  _,A_:%_76_7:a Aaafpc Bt|j|||||||S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )r~read)rrrrr r!r"r#s r}from_fprs/      pathc jt|d5}t||||||||cdddS#1swYyxYw)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr) rrrrr r!r"r#rs r} from_pathrs< dD R            s)2c t|||||||}t|}tt|} t |dk(rt dj ||j} | dxxd| jzz cc<tdj t|j|dj| d5} | j| jddd| S#1swY| SxYw)zi Take a (text-based) file path and try to create another file next to it, this time using UTF-8. rz;Unable to normalize "{}", no encoding charset seems to fit.-z{}r'wbN)rrlistrrBIOErrorr;bestr+rrMreplacerDwriteoutput) rrrrr r!r"rafilenametarget_extensionsresultrs r} normalizers  G~HXh/0 7|q I P P   \\^FaC&//11  CI%%h8I0JKLd   !  M   Ms  C00C:)皙?NNTF)rrrNNT)3os.pathrrtypingrrrrosr rKrMr?cdr r r rconstantrrrmdrmodelsrrutilsrrrrrr getLoggerr=r>DEBUG StreamHandlerhandler setFormatter Formatter addHandlerr9rEfloatboolr~rrrrr}rs&00 KJ0   / 0  '   !&W&&'RST' ""!%JJ JJ J s) J s) JJJJ^ ""!%   s)  s) 8""!%      s)  s)     :""!%) ) )) ) s) ) s) )))CHsEEE