dZddlmZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l m Z dd l mZdd l mZd ZGd dej"Zej&j(dej&j*dej&j,dej&j.dej&j0dej&j2diZdZddZdZdZeeddddfdZdZ dZ!GddZ"y)z3Used to collect anonymous transfer-related metrics.)absolute_import)division)unicode_literalsN) storage_url) task_util)metrics) properties)files) platformscttjjjj rtjj j jjdd}tjjtjj j |dyy)aETransform the user agent when the gsutil shim is used to run gcloud. This transforms `gcloud.storage.command` to `gcloud.gsutil.command`. This needs to be called by every command, so the best place to put this is likely surface/storage/__init__.py, but if there's a better place it could be called there instead. zgcloud.storage.zgcloud.gslibshim.N) r VALUESstoragerun_by_gsutil_shimGetBoolr command_nameGetreplaceSetInvocationValue)command_path_strings 6lib/googlecloudsdk/command_lib/storage/metrics_util.pyfix_user_agent_for_gsutil_shimrs1199;$++33@@DDFNN.0((!!..0CTK<ceZdZdZdZy)ParallelismStrategyN)__name__ __module__ __qualname__ SEQUENTIALPARALLELrrrr/s * (rrrrctjjjj }tj |d|z|y)zCommon code for processing an event. Args: metric (str): The metric being recorded. value (mixed): The value being recorded. zStorage-N)r r rrrCustomKeyValue)metricvaluers r_record_storage_eventr+As;""**77;;=, zF':EBrctjrtjjStj jSN)rshould_use_parallelismrr!r*r r"rr_get_parallelism_strategyr/Ls2%%'  ' ' - --  ' ' - --rcntjjjj rdSdS)Nrr)r r rrrr"rr_get_run_by_gsutil_shimr1Rs*''::BBDK!Krctdttdttd|td|td|td|td|td|y ) aReports metrics for a transfer. Args: source_scheme (int): The source scheme, i.e. 'gs' or 's3'. destination_scheme (int): The destination scheme i.e. 'gs' or 's3'. num_files (int): The number of files transferred. size (int): The size of the files transferred, in bytes. avg_speed (int): The average throughput of a transfer in bytes/sec. disk_io_time (int): The time spent on disk of a transfer in ms. rRunByGsutilShim SourceSchemeDestinationScheme NumberOfFilesSizeOfFilesBytesAverageSpeedBytesPerSec DiskIoTimeMsN)r+r/r1 source_schemedestination_scheme num_filessize avg_speed disk_io_times rreportrAVsb-/H/JK)+B+DE 6+-?@3*D119= 5rc@g} tjd5}|jdd}|D]<}|j\}}}}|dj s,|j |> ddd|S#1swY|SxYw#tj $rY|SwxYw)z_Retrieves a list of disk partitions. Returns: An array of partition names as strings. z/proc/partitionsrN)r FileReader readlinessplitisdigitappendError) partitionsflinesline_names r_get_partitionsrPls *    , -kkmABe$ 1a 8      D ! .  .     s5BAA9A9/B9B>BBBBc d}t}i} tjd5}|j}|D]{}|j dd}|\ }}}} }} } } }} }||vs)t | |z} t | |z} t | } t | } t | } t |}| | | | | |f||<} ddd|S#1swY|SxYw#tj $rY|SwxYw)aRetrieves disk I/O statistics for all disks. Adapted from the psutil module's psutil._pslinux.disk_io_counters: http://code.google.com/p/psutil/source/browse/trunk/psutil/_pslinux.py Originally distributed under under a BSD license. Original Copyright (c) 2009, Jay Loden, Dave Daeschler, Giampaolo Rodola. Returns: A dictionary containing disk names mapped to the disk counters from /disk/diskstats. iz/proc/diskstatsN )rPr rDrErFintrI) sector_sizerJretdictrKrLrMvaluesrNrOreadsrbytesrtimewriteswbyteswtimes rget_disk_countersr]s + * '   + ,kkme$cr"HNE1dE1feVQ : v;,&v;,&e*%v;&e*%e*% &&&%G'$- -$ .% -$ .   . s5C;C!AC7CC C CC%$C%c.eZdZdZdZdZdZdZdZy)MetricsReporterz9Mix-in for tracking metrics during task status reporting.cNt|_t|_t|_yr-)UNSET_source_scheme_destination_schemer]_disk_counters_start)selfs r__init__zMetricsReporter.__init__sD$D 1 3Drc8|rt|jStS)z;Extracts the scheme as an integer value from a storage_url.)PROVIDER_PREFIX_TO_METRICS_KEYschemera)reurls r_get_scheme_valuez!MetricsReporter._get_scheme_values +CJJ 77 Lrc|jtk(r |j|j|_|jtk(r!|j|j |_yy)zSets source and destination schemes, if available. Args: status_message (thread_messages.*): Message to process. N)rbrark source_urlrcdestination_url)restatus_messages r#_set_source_and_destination_schemesz3MetricsReporter._set_source_and_destination_schemessZ  e# 22>3L3LMd 5(!%!7!7  ( ("*d)rc tjj}|tjjk(rw|j}t }t |jDcgc] }|d|dzc}t |jDcgc] }|d|dzc}z StScc}wcc}w)z&Calculate deltas of time spent on I/O.r$r%) r OperatingSystemCurrentLINUXrdr]sumrVra)re current_os disk_startdisk_endstats r_calculate_disk_ioz"MetricsReporter._calculate_disk_ios**224JY..444,,j"$h1BC1B47T!W$1BCD1B1B1DE1D47T!W$1DEFGH LDEs )B9B> c tt|t|z }t|j|j||||j y)zReports back all tracked events via report method. Args: total_bytes (int): Amount of data transferred in bytes. time_delta (int): Time elapsed during the transfer in seconds. num_files (int): Number of files processed r:N)roundfloatrArbrcrz)re total_bytes time_deltar=r?s r_report_metricszMetricsReporter._report_metricssLeK(5+<<=I ))33 ,,. 0rN) rrr__doc__rfrkrprzrr"rrr_r_sA4 * 0rr_)r)#r __future__rrrenum"googlecloudsdk.command_lib.storager(googlecloudsdk.command_lib.storage.tasksrgooglecloudsdk.corerr googlecloudsdk.core.utilr r rEnumrProviderPrefixFILEGCSHTTPHTTPSPOSIXS3rhrar+r/r1rArPr]r_r"rrrs:&' :>'**. K $)) ##Q""A##Q$$a$$a!!1 " C. L5AQQ6,.(V7070r