q d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlZddl Z ddl
Z
ddl Z ddl Z ddl
Z
ddlZddlZddlZddlZddlmZ ddlZdd lmZ dd
lmZ dd lmZ dd lmZ dd
l m Z ddl m Z ddl m!Z! ddl m"Z" ddl m#Z# ddl$m%Z% ddl$m&Z& ddl'm(Z( ddl)Z)ejV re,Z-dZ.dZ/dZ0dZ1de1z de.z z Z2 edd Z3d Z4d Z5d Z6d Z7d Z8 d$d!Z9 G d" d#e Z:y# e*$ r dZ)Y Sw xY w)%&Implementation of gsutil test command. )absolute_import)print_function)division)unicode_literals)
namedtupleN)range)ProjectIdException)Command)ResetFailureCount)CommandException)PopulateProjectId) GetTestNames)InvokedFromParFile)unittest)NO_MAX)UTF8)
IS_WINDOWS 2 sequential_onlyz3
gsutil test [-l] [-u] [-f] [command command...]
z
SYNOPSIS
a
DESCRIPTION
The gsutil test command runs the gsutil unit tests and integration tests.
The unit tests use an in-memory mock storage service implementation, while
the integration tests send requests to the production service using the
`preferred API
`_ set in the
boto configuration file.
CAUTION: The ``test`` command creates test buckets and objects in your project.
Force quitting the ``test`` command can leave behind stale buckets, objects,
and HMAC keys in your project.
To run both the unit tests and integration tests, run the command with no
arguments:
gsutil test
To run the unit tests only (which run quickly):
gsutil test -u
Tests run in parallel regardless of whether the top-level -m flag is
present. To limit the number of tests run in parallel to 10 at a time:
gsutil test -p 10
To force tests to run sequentially:
gsutil test -p 1
To have sequentially-run tests stop running immediately when an error occurs:
gsutil test -f
To run tests for one or more individual commands add those commands as
arguments. For example, the following command will run the cp and mv command
tests:
gsutil test cp mv
To list available tests, run the test command with the -l argument:
gsutil test -l
The tests are defined in the code under the gslib/tests module. Each test
file is of the format test_[name].py where [name] is the test name you can
pass to this command. For example, running "gsutil test ls" would run the
tests in "gslib/tests/test_ls.py".
You can also run an individual test class or function name by passing the
test module followed by the class name and optionally a test name. For
example, to run the an entire test class by name:
gsutil test naming.GsutilNamingTests
or an individual test function:
gsutil test cp.TestCp.test_streaming
You can list the available tests under a module or class by passing arguments
with the -l option. For example, to list all available test functions in the
cp module:
gsutil test -l cp
To output test coverage:
gsutil test -c -p 500
coverage html
This will output an HTML report to a directory named 'htmlcov'.
Test coverage is compatible with v4.1 of the coverage module
(https://pypi.python.org/pypi/coverage).
OPTIONS
-b Run tests against multi-regional US buckets. By default,
tests run against regional buckets in us-central1.
-c Output coverage information.
-f Exit on first sequential test failure.
-l List available tests.
-p N Run at most N tests in parallel. The default value is %d.
-s Run tests against S3 instead of GS.
-u Only run unit tests.
TestProcessDataz name return_code stdout stderrc B G fddt j S )zCreates a closure of CustomTestResult.
Args:
total_tests: The total number of tests being run.
Returns:
An instance of CustomTestResult.
c ( e Zd ZdZ fdZ xZS )3MakeCustomTestResultClass..CustomTestResultzDA subclass of unittest.TextTestResult that prints a progress report.c t | | | j rdj |j j d dd }d| j t | j t | j t | j |fz }|d d }|j d }| j j d|z y y )N.z(
%d/%d finished - E[%d] F[%d] s[%d] - %sI z%s - )super startTestdotsjoinidsplittestsRunlenerrorsfailuresskippedljuststreamwrite)selftesttest_idmessageCustomTestResult __class__ total_testss &platform/gsutil/gslib/commands/test.pyr! z=MakeCustomTestResultClass..CustomTestResult.startTest s d-d3
((4779??3/45>MM;DKK0@
&DLL(97DD #2,--# 'G+,
)__name__
__module__ __qualname____doc__r!
__classcell__)r3 r2 r4 s @r5 r2 r s N - -r6 r2 )r TextTestResult)r4 r2 s `@r5 MakeCustomTestResultClassr= s -00 -
r6 c | g}g }|rq|j }|D ]Y }t |t j r|j | /|j |j t
d d [ |rq|S )zGTakes a list of test suites and returns a list of contained test names.gslib.tests.test_N)pop
isinstancer TestSuiteappendr$ r' )
test_suitesuites
test_namessuiter/ s r5 GetTestNamesFromSuitesrH sq <&* JJLE D(,, -
d$'')C(;$<$=>?
r6 c p t | j j d d dz | j z S )z%>EK{ 0%8 N9 56
["5t
< nY 78
Y 3T
: nY 78 I 1 2 &&~i'@A&--nY.GH % !
"F>$:
*
+
1
2
4 4r6 c ( d}| D ]
}|r|dz
} |S )z*Counts number of falses in the input list.r rK )
input_list num_falser` s r5 CountFalseInListrg s% )d 1ni
r6 c , |}t st j gng }t j j
rdgng }t j j rdgng } g }
ddt z g}
t j } | } t | |k r|t | k rt j j }
|r||
d< t }t! |t" j$ gz |
z dgz |z | z dt& z gz | | t d d gz D cg c] }t) j* | }}t) j, |
D ]0 \ }}t) j* | |t) j* | < 2 |j/ t1 j2 |t0 j4 t0 j4 |
|d z
}|j/ d t j | z
d
kD r:t7 d||z
t | t | fz t j } t | |k r|t | k r|t | k( r"t7 dt | t | fz |S # t $ r Y w xY wc c}w )ag Creates test processes to run tests in parallel.
Args:
parallel_tests: List of all parallel tests.
test_index: List index of last created test before this function call.
process_list: List of running subprocesses. Created processes are appended
to this list.
process_done: List of booleans indicating process completion. One 'False'
will be added per process created.
max_parallel_tests: Maximum number of tests to run in parallel.
root_coverage_file: The root .coverage filename if coverage is requested.
Returns:
Index of last created test.
-s-bz-oz GSUtil:default_project_id=%sGSUTIL_COVERAGE_OUTPUT_FILEr/ --r? N)stdoutstderrenvrK Fr z.Created %d new processes (total %d/%d created)z.Test process creation finished (%d/%d created)) r sys
executabletestsutil RUN_S3_TESTSUSE_MULTIREGIONAL_BUCKETSr r
timerg r' osenvironcopydictlistgslib GSUTIL_PATH_SEQUENTIAL_ISOLATION_FLAGsix
ensure_str iteritemsrC
subprocessPopenPIPEprint)parallel_tests
test_index process_list process_donemax_parallel_testsroot_coverage_fileorig_test_indexexecutable_prefix s3_argumentmultiregional_bucketsproject_id_argprocess_create_start_time
last_log_timero envstrpartcmdkvs r5 CreateTestProcessesr s * / /A.Bs~~& **11r+$)JJ$H$H4&b. ,/@/BBN #iik+- , '*< < c.) )
**//
C+=c
'(
VF *.
H
"
" .
. /
0 J
',?(@(A
B C
D *
*
t *
c"1"%.."3fS^^A
# * * # %&
!OJ yy{]"Q&
=?*C ,=.!## % iikmA
, '*< < c.) )@ 3~&& ; s> 2 3 4 6 U
s !J J J
Jc e Zd ZdZ ej
dg ededdddeg
Z ej dg dd e i
Z d Z
d Z e e fd
Zd Zy) TestCommandr r/ r zbuflp:scTF) command_name_aliasesusage_synopsismin_argsmax_argssupported_sub_args file_url_okprovider_url_okurls_start_argsupported_private_args command_helpz2Run gsutil unit/integration tests (for developers)) help_namehelp_name_aliases help_typehelp_one_line_summary help_textsubcommand_help_textc g }g }g }d}d}d} t | }
t j x} } t |d|||| }
t | |
k rt t | D ] }|| s|| j d||< || j \ }}|| j j || j j || j dk7 r|dz
}|j t || || j || t | |
k rt ||
|||| }
t | |
k rt j | z
dkD rt dt | |
|fz t | | k( r|dz
}n
t | } d}|dkD rzg }t t | D ] }|| r |j || t j | z
}t d j t t | |
t j } t j d t | |
k rt j }|r}|D ]x }|j" dk7 s|j j% d }t d |j& z |D ]. }t |j) t* j- 0 z ||| z
fS )
aZ Executes the parallel/isolated portion of the test suite.
Args:
parallel_integration_tests: List of tests to execute.
max_parallel_tests: Maximum number of parallel tests to run at once.
coverage_filename: If not None, filename for coverage output.
Returns:
(int number of test failures, float elapsed time)
r )r TrK )name return_coderm rn r z %d/%d finished - %d failures zL{sec} seconds elapsed since beginning parallel tests.
Still running: {procs})secprocs
zResults for failed test %s:)r' rv r r poll communicaterm closern
returncoderC r r formatrL intsleepr r% r decoder strip)r. parallel_integration_testsr coverage_filenamer r process_resultsnum_parallel_failures progress_less_logging_cyclescompleted_as_of_last_lognum_parallel_testsparallel_start_timer r proc_numrm rn
still_runningelapsedprocess_run_finish_timeresult
new_stderrlines r5 RunParallelTestsz TestCommand.RunParallelTests s LLO #$ 78*.))+5-$%?%&%1%1%78I KJ o
!3
3C -.( !\(%;%@%@%B%J
!% X %h/;;=X %%++-X %%++- ! , , 1
1
$
!;H!E(4X(>(I(I#)#)
+ , /
\ / /()C)3)5)5);> ? u%< = U#8 9 ; ;= -. G %7 Dk
'(
)* r6 c
t | dkD st | dkD rgt | }t | }|j | }t d |D ]/ \ }}t d|j z t d|z 1 yy)a Prints all skipped tests, and the reasons they were skipped.
Takes the union of sequentual_skipped and parallel_skipped,
and pretty-prints the resulting methods and reasons. Note that these two
arguments are lists of tuples from TestResult.skipped as described here:
https://docs.python.org/2/library/unittest.html#unittest.TestResult.skipped
Args:
sequentual_skipped: An instance of TestResult.skipped.
parallel_skipped: An instance of TestResult.skipped.
r zTests skipped:z z Reason: N)r' setunionr r$ )r. r parallel_skipped all_skippedmethodreasons r5 r z TestCommand.PrintSkippedTests s "c*:&;a&? 12 -.&,,-=>k '.&&
dVYY[ !
nv%& (
'@r6 c d}d}t }d}d}| j r| j D ] \ }}|dk( rdt j _ !|dk( rd})|dk( rd}1|dk( rd}9|dt
z k( rd}H|dk( r t
| }Y|d k( r;t j j s t d
dt j _ |d k( sdt j _
|rt s t d t j j r/|t kD r&| j j d
t t }t t! }|rH| j" st/ j4 } | D ]% }
| j7 |
}| j9 | ' |rGtM }t% dt' | z t% ddj) t | ytO jP jS tN jT k rd}n%d}tO jV tN jX |rPt j d gd d t[ j\ d z g }|j_ |ja d}d}tc \ }}}}t j j sg }tO jd d!| tO jd d"| tO jd d#| tO jd d$| |xs t' | dk xr | }d%tf jh d&< |rV| jk } tm | } t/ jn || |' } | jq | } | js }n|dk( r||z
}g }tu jt } t' | t' | z rt% d(t' | t' | z z ||z }!| jw t |!D "cg c] }"|" c}" } | jk }#tm |# } t/ jn || |' } | jq | } | js }| jx }$nd}#d}tu jt | z
}%||z
}t' | }&|&sng }$t{ ||& }'|&dkD rH|dkD rCd)}(|'t kD r|(d*z
}(t% d+j) t} j~ |(|&|'fz nt% d,|&z | j |||rj j nd \ }})| j |#||$|%|&||) |rRj |j |j t% d-|j j z d.tf jh d&< |r
|s t yyc c}
w # t: t< f$ r}d|
dt? | } tA |
t | # tB $ rG}tE jF }tI jJ dd| }|d|z z
}Y d}~t | d}~ww xY wd}~ww xY wc c}"w )/z)Command entry point for the test command.Frj Tz-cz-fz-lrl z-pri zbS3 tests require S3 credentials. Please add appropriate credentials to your .boto file and re-run.z-uzrCoverage has been requested but the coverage module was not found. You can install it with "pip install coverage".zCReducing parallel tests to %d due to S3 maximum bucket limitations.zFound %d test names: z
r r zgslib.tests.test_%sz%Failed to import test code from file z#. TestLoader provided this error:
z\nz
z
Additional traceback:
%sNrK r| zgslib/third_party/*z
gslib/tests/**)sourceomitzSequential tests to run: %szIsolated tests to run: %sz Parallel unit tests to run: %sz%Parallel integration tests to run: %s1GSUTIL_TEST_ANALYTICS) verbosity resultclassfailfastz Running %d tests sequentially.z1Running %d tests in parallel mode (%d processes).z Please be patient while your CPU is incinerated. If your machine becomes unresponsive, consider reducing the amount of parallel test processes by running 'gsutil test -p '.
z4Running %d tests sequentially in isolated processes.z%Coverage information was saved to: %s0)H _DEFAULT_TEST_PARALLEL_PROCESSESsub_optsrr rs ru r~ long HAS_S3_CREDSr
rt RUN_INTEGRATION_TESTScoverage#_DEFAULT_S3_TEST_PARALLEL_PROCESSESloggerwarnrX r argsr r' r# r% rC r installHandler
TestLoaderrB loadTestsFromNameaddTests ImportErrorAttributeErrorrL
__import__ Exception traceback
format_excresubrH logging getLoggergetEffectiveLevelINFOdisableERRORtempfile
gettempdirerasestartrb debugrw rx countTestCasesr= TextTestRunnerrun
wasSuccessfulrv loadTestsFromNamesr* mintextwrapwrapr
data_filesfilenamer stopcombinesaver )*r. r
list_testsr perform_coverager oarF commands_to_testr loaderrG command_namesuite_for_current_commandemsg stack_traceerrr coverage_controllerr r rZ rY parallel_unit_testsr run_tests_sequentiallyr4 r runnerretsequential_start_timesequential_tests_to_run test_namer r r r
num_processesr1 r s* r5
RunCommandzTestCommand.RunCommand
s% HJ9 O }}--$!Q 915%**
.
$Y !
$Y(
$Y*
444
5 !/
$Y #Aw
$Y((" $6 7
7 %)%**
!
$Y-2%**
*- 0 <
= =
@@
kk =? ? 'J$)) "S_ 45 CVJ /01
yy))$ : C !3z!A
!
!"7$">
?
!
!$
' DNN:4/$6:N
"F "e*, &&,&>&>|&L
#
..2
3 +( )%0j "S_ 45 CVJ /01
,,.',,>ii
oogmm$
% - -gY2G2A2:2E2E2G#2M40 1 ! ! %A$G!~': :: + +#% MM/1AB MM-~> MM24GH MM9,. . 3 "#= >! C 2#1 1 +.BJJ&' ((*k-k:k&&3>08:f
JJu c ,,. q 66%'" "iik
%8!9 9
.#$s+>'??A B"25H"H )) /FG/F)I/FG HJ $ 3 3 5 /0DE ((95@2:< jj . . 0 [[ ! $ .C C !N2 9:
.0BC
!&81&<G'
=
= 6
7G
mmG'9=&I IJL N G#$ &9=9N9N &(:
* * 3 3%):+6 5
24F 02I 02G 3 5
!!#
4
+
+
4
45 7 +.BJJ&'"7
Q O ^, &(4c!f># = | $ !%
% =#..0K&&+6C 4< # )X=t+4 25 ',c
r6 r )N);r:
__future__r r r r collectionsr r rw r r rp r r rv r r six.movesr r| gslib.cloud_apir
gslib.commandr r gslib.exceptionr
gslib.project_idr gslib.testsrr gslib.tests.utilr r r gslib.utils.constantsr r gslib.utils.system_utilr r r PY3r r r r r~ r. r1 r r= rH rO rb rg r r rd r6 r5 r? s - & % ' "
. ! + , . ) / % ( & . 77 $#$ &( # .
^| '}^'`' D . =?6 %24j ,0 K\H
' H
g
(s C* *C43C4