g;dZddlZddlZddlZddlZddlZddlmZddl Z ddl m Z ddl m Z GddejjZGdd ejjZGd d ejjZGd d ejjZGddejjZGddejjZGddejjZGddeZejj0j3eeGddeZejj0j3eey)z0Classes for dealing with I/O from ML pipelines. N) cStringIO) _decoders)_filec"eZdZdZdZdZdZy)ExampleProtoCoderz8A coder to encode and decode TensorFlow Example objects.c.ddl}|j|_yNr) tensorflowtrain _tf_train)selftfs ,lib/third_party/ml_sdk/cloud/ml/io/coders.py__init__zExampleProtoCoder.__init__"sXXDNc"|jS)zEncodes Tensorflow example object to a serialized string. Args: example_proto: A Tensorflow Example object Returns: String. )SerializeToString)r example_protos rencodezExampleProtoCoder.encode&s  * * ,,rc\|jj}|j||S)zDecodes a serialized string into a Tensorflow Example object. Args: serialized_str: string Returns: Tensorflow Example object. )r ExampleParseFromString)r serialized_strexamples rdecodezExampleProtoCoder.decode1s)nn$$&G N+ NrN__name__ __module__ __qualname____doc__rrrrrrrs@ - rrc$eZdZdZddZdZdZy) JsonCoderz1A coder to encode and decode JSON formatted data.Nc||_yN)_indent)r indents rrzJsonCoder.__init__Bs DLrcFtj||jdS)zvEncodes a python object into a JSON string. Args: obj: python object. Returns: JSON string. ),z: )r' separators)jsondumpsr&r objs rrzJsonCoder.encodeEs ::c$,,; GGrc,tj|S)zDecodes a JSON string to a python object. Args: json_string: A JSON string. Returns: A python object. )r+loads)r json_strings rrzJsonCoder.decodeQs ::k ""rr%rr!rrr#r#?s9 H #rr#c@eZdZdZGddeZ ddZdZdZy) CsvCoderz3A coder to encode and decode CSV formatted data. c(eZdZdZdZdZdZdZy)CsvCoder._WriterWrapperz?A wrapper for csv.writer / csv.DictWriter to make it picklable.c|||f|_t|_|r)tj|j|d||_ytj |jd||_y)N)lineterminator delimiter)_stater_buffercsv DictWriter_writerwriter)r column_namesr9decode_to_dicts rrz CsvCoder._WriterWrapper.__init__ds[!9n=dk[dl ~~ LL  ! zz LL! rc|jj||jj}|jj d|jj d|Sr )r>writerowr;getvalueseektruncate)r recordvalues r encode_recordz%CsvCoder._WriterWrapper.encode_recordssO llF#ll##%e ll llA lrc|jSr%)r:r s r __getstate__z$CsvCoder._WriterWrapper.__getstate__{s [[rc"|j|yr%)r)r states r __setstate__z$CsvCoder._WriterWrapper.__setstate__~sdmmUrN)rrrr rrIrLrOr!rr_WriterWrapperr5asI !rrPcttj|||||||_|j||||_y)aInitializes CsvCoder. Args: column_names: Tuple of strings. Order must match the order in the file. numeric_column_names: Tuple of strings. Contains column names that are numeric. Every name in numeric_column_names must also be in column_names. delimiter: A one-character string used to separate fields. decode_to_dict: Boolean indicating whether the docoder should generate a dictionary instead of a raw sequence. True by default. fail_on_error: Whether to fail if a corrupt row is found. Default is True. skip_initial_space: When True, whitespace immediately following the delimiter is ignored when reading. )r@r9rAN)r CsvDecoder_decoderrP_encoder)r r@numeric_column_namesr9rA fail_on_errorskip_initial_spaces rrzCsvCoder.__init__sH"((*I~)+DM''!%('DMrc8|jj|S)aBDecode csv line into a python dict. Args: csv_line: String. One csv line from the file. Returns: Python dict where the keys are the column names from the file. The dict values are strings or numbers depending if a column name was listed in numeric_column_names. Missing string columns have the value '', while missing numeric columns have the value None. If there is an error in parsing csv_line, a python dict is returned where every value is '' or None. Raises: Exception: The number of columns to not match. )rSr)r csv_lines rrzCsvCoder.decodes" ==   ))rc8|jj|S)aSEncode python dict to a csv-formatted string. Args: python_data: A python collection, depending on the value of decode_to_dict it will be a python dictionary where the keys are the column names or a sequence. Returns: A csv-formatted string. The order of the columns is given by column_names. )rTrI)r python_datas rrzCsvCoder.encodes == & &{ 33rN)r)TTF) rrrr objectrPrrrr!rrr3r3]s0v@DG26"''2*& 4rr3c"eZdZdZdZdZdZy) YamlCoderz1A coder to encode and decode YAML formatted data.ctjr+tj|_tj|_yt jdtj|_tj|_y)zTrying to use the efficient libyaml library to encode and decode yaml. If libyaml is not available than we fallback to use the native yaml library, use with caution; it is far less efficient, uses excessive memory, and leaks memory. zpCan't find libyaml so it is not used for YamlCoder, the implementation used is far slower and has a memory leak.N) yaml__with_libyaml__ CSafeDumper _safe_dumper CSafeLoader _safe_loaderloggingwarning SafeDumper SafeLoaderrKs rrzYamlCoder.__init__sV **d**d oo EF//d//drcHtj|dd|jS)zvEncodes a python object into a YAML string. Args: obj: python object. Returns: YAML string. Fzutf-8)default_flow_styleencodingDumper)r`dumprcr-s rrzYamlCoder.encodes' 99     ""rcDtj||jS)zDecodes a YAML string to a python object. Args: yaml_string: A YAML string. Returns: A python object. )Loader)r`loadre)r yaml_strings rrzYamlCoder.decodes 99[):): ;;rNrr!rrr^r^s9*$ " >r) rrrr rrrpropertyrrrrrr!rrrrpsw ! 2 2 22 ># ##=+~   ?rrc(eZdZdZdZdZdZdZy)rzResult of training a model.c<d|_d|_d|_d|_yr%rtraining_job_metadataerrorrrKs rrzTrainingJobResult.__init__s% D"&D DJ#DrcHdD]}t||t||k7syy)NrFTrrs rrzTrainingJobResult.__eq__s+% q WQ] *% rc||k( Sr%r!rs rrzTrainingJobResult.__ne__rrcg}|jjD]&\}}|jdj||(djdj |S)NrzTrainingJobResult({0})rrrs rrzTrainingJobResult.__repr__sU F '')1 mmI$$Q*+* # * *499V+< ==rN)rrrr rrrrr!rrrrs# $>rr)r r<rr+rf apache_beambeam six.movesrr`google.cloud.ml.utilrrcodersCoderrr#r3r^rtrrr\rregistryregister_coderrr!rrrs!   *& ))@# !!#<[4t{{  [4|-< !!-<`.1DKK%%.1b' dkk//' T' T[[..' T_?_?D ##$68OP > >F ##$57MNr