+h NSSKJr SSKJrJrJr SSKJrJrJ r "SS\5r g)) annotations)AnyOptionalcast) TextSplitter Tokenizersplit_text_on_tokensc^\rSrSr%SrS S U4SjjjrSSjrSSjrSSjrSr S\ S 'SS jr S r U=r $)%SentenceTransformersTokenTextSplitterz8Splitting text to tokens using sentence model tokenizer.intc  >[TU]"S0UDSU0D6 SSKJn X lU"UR 5UlUR RUlURUS9 g![anSn[ U5UeSnAff=f)zCreate a new TextSplitter. chunk_overlapr)SentenceTransformerzCould not import sentence_transformers python package. This is needed in order to for SentenceTransformersTokenTextSplitter. Please install it with `pip install sentence-transformers`.N)tokens_per_chunk) super__init__sentence_transformersr ImportError model_name_model tokenizer_initialize_chunk_configuration) selfrrrkwargsrerrmsg __class__s ^/root/34ku/venv/lib/python3.13/site-packages/langchain_text_splitters/sentence_transformers.pyr.SentenceTransformersTokenTextSplitter.__init__ s ?6?? , A%)$//: .. ,,>N,O ,N  c" +  ,sA%% B/A==BcURRUlUcURUlOXlURUR:a5SURSURSURS3n[ U5eg)NzThe token limit of the models 'z' is: z. Argument tokens_per_chunk=z > maximum token limit.)rmax_seq_lengthmaximum_tokens_per_chunkrr ValueError)rrrs r rESentenceTransformersTokenTextSplitter._initialize_chunk_configuration$s)- (B(B%  #$($A$AD !$4 !  4#@#@ @1$//1BC5567..2.C.C-D)+  S/ ! Ac^SU4Sjjn[TRTRTRRUS9n[ XS9$)aSplits the input text into smaller components by splitting text on tokens. This method encodes the input text using a private `_encode` method, then strips the start and stop token IDs from the encoded result. It returns the processed segments as a list of strings. Args: text (str): The input text to be split. Returns: List[str]: A list of string components derived from the input text after encoding and processing. c,>TRU5SS$)N)_encode)textrs r %encode_strip_start_and_stop_token_ids_SentenceTransformersTokenTextSplitter.split_text..encode_strip_start_and_stop_token_idsFs<<%a+ +r')rrdecodeencode)r-rr-strreturn list[int])r_chunk_overlaprrr0r )rr-r.rs` r split_text0SentenceTransformersTokenTextSplitter.split_text7sC ,--!22>>((8  $CCr'c6[URU55$)aACounts the number of tokens in the given text. This method encodes the input text using a private `_encode` method and calculates the total number of tokens in the encoded result. Args: text (str): The input text for which the token count is calculated. Returns: int: The number of tokens in the encoded text. )lenr,)rr-s r count_tokens2SentenceTransformersTokenTextSplitter.count_tokensRs4<<%&&r'l _max_length_equal_32_bit_integercdURRUURSS9n[SU5$)Ndo_not_truncate) max_length truncationr5)rr1r=r)rr-&token_ids_with_start_and_end_token_idss r r,-SentenceTransformersTokenTextSplitter._encodebs<151F1F <<(2G2 . K!GHHr')rr$rrr)2z'sentence-transformers/all-mpnet-base-v2N) rr rr3r Optional[int]rrr4None)rrEr4rF)r-r3r4z list[str])r-r3r4r r2)__name__ __module__ __qualname____firstlineno____doc__rrr7r;r=__annotations__r,__static_attributes__ __classcell__)rs@r r r sB C*. PPP( P  P  PP2"#0" "&D6 '-2$c1IIr'r N) __future__rtypingrrrlangchain_text_splitters.baserrr r rr'r rRs$"&&WW`IL`Ir'