xdZddlZddlZddlZGddeZGddeZGddeZGd d eZ y) zDataflow-related utilities. NceZdZdZy) DecodeErrorzBase decode error.N)__name__ __module__ __qualname____doc__1lib/third_party/ml_sdk/cloud/ml/util/_decoders.pyrrsr rceZdZdZy)PassthroughDecoderc|SNr selfxs r decodezPassthroughDecoder.decodes Hr N)rrrrr r r r r s r r ceZdZdZdZy) JsonDecoderz"A decoder for JSON formatted data.c,tj|Sr)jsonloadsrs r rzJsonDecoder.decode$s ::a=r N)rrrrrr r r rr!s *r rcTeZdZdZGddeZGddeZdZdZdZ d Z y ) CsvDecoderz$A decoder for CSV formatted data. c(eZdZdZdZdZdZdZy)CsvDecoder._LineGeneratorzCA csv line generator that allows feeding lines to a csv.DictReader.cg|_yr)_linesrs r __init__z"CsvDecoder._LineGenerator.__init__1s dkr cV|jrJ|jj|yr)rappend)rlines r push_linez#CsvDecoder._LineGenerator.push_line4s _ kkr c|Srr rs r __iter__z"CsvDecoder._LineGenerator.__iter__9s kr ct|j}|dk(r td|dk(s Jd|z|jjS)Nrz@Columns do not match specified csv headers: empty line was foundzUnexpected number of lines %s)lenrrpop)r line_lengths r nextzCsvDecoder._LineGenerator.next<sX $k   NP P A L>LL [[__ r N)rrrrr r$r&r,r r r _LineGeneratorr.sM  r r-c(eZdZdZdZdZdZdZy)CsvDecoder._ReaderWrapperz?A wrapper for csv.reader / csv.DictReader to make it picklable.c|||||f|_||_|r(tj||t |||_ytj |t |||_y)N) delimiterskipinitialspace)_state_line_generatorcsv DictReaderstr_readerreader)rline_generator column_namesr1decode_to_dictskip_initial_spaces r r z"CsvDecoder._ReaderWrapper.__init__Ms^#\9n')dk+d ~~ LC N/1 zz.C N3EG r cl|jj||jjSr)r4r$r8r,rs r read_recordz%CsvDecoder._ReaderWrapper.read_recordZs) $$Q' \\    r c|jSr)r3rs r __getstate__z&CsvDecoder._ReaderWrapper.__getstate__^s [[r c"|j|yr)r )rstates r __setstate__z&CsvDecoder._ReaderWrapper.__setstate__asdmmUr N)rrrrr r?rArDr r r _ReaderWrapperr/JsI G!r rEc||_t||_|j|j |||||_||_||_y)aInitializer. Args: column_names: Tuple of strings. Order must match the order in the file. numeric_column_names: Tuple of strings. Contains column names that are numeric. Every name in numeric_column_names must also be in column_names. delimiter: String used to separate fields. decode_to_dict: Boolean indicating whether the docoder should generate a dictionary instead of a raw sequence. True by default. fail_on_error: Whether to fail if a corrupt row is found. skip_initial_space: When True, whitespace immediately following the delimiter is ignored. N) _column_namesset_numeric_column_namesrEr-r8_decode_to_dict_fail_on_error)rr;numeric_column_namesr1r< fail_on_errorr=s r r zCsvDecoder.__init__dsS"&D!$%9!:D&& |YDL*D'Dr c^|jr t|tjd|y)ajHandle corrupt rows. Depending on whether the decoder is configured to fail on error it will raise a DecodeError or return None. Args: message: String, the error message to raise. Returns: None, when the decoder is not configured to fail on error. Raises: DecodeError: when the decoder is configured to fail on error. zDiscarding invalid row: %sN)rKrloggingwarning)rmessages r _handle_corrupt_rowzCsvDecoder._handle_corrupt_row}s,    oo2G< r c^|r|jsy||jvr t|S|Sr)striprIfloat)r column_namevalues r _get_valuezCsvDecoder._get_values/  d000 5\ Lr cd |jj|}t |t |j k7r!|jd|j d|S|j rb||j d!|jd|j d|S|jD]\}}|j||||<|St|j D]\}}||}|j||||<!|S#t$r }|j|d|cYd}~Sd}~wwxYw)axDecodes the given string. Args: record: String to be decoded. Returns: Serialized object corresponding to decoded string. Or None if there's an error and the decoder is configured not to fail on error. Raises: DecodeError: If columns do not match specified csv headers. ValueError: If some numeric column has non-numeric data. z: Nz,Columns do not match specified csv headers: z -> ) r8r? ExceptionrRr)rGrJ iteritemsrX enumerate)rrecordenamerWindexs r rzCsvDecoder.decodes8>||''/f 6{c$,,--  % %  & *++  ""2& ' /''""F ,- - ))++$tU3t , M#4#5#56+%u e4u 7 M- >  % %!V&< ==>sD D/D*$D/*D/N) rrrrobjectr-rEr rRrXrr r r rr(s3 v8v4(2(&r r) rr5rrOr[rrbr rrr r r rcsG )   &[[r