Tmcnulty at 18:30, 30 March 2008

2008-03-30T18:30:25Z

← Older revision		Revision as of 18:30, 30 March 2008
Line 54:		Line 54:

	===show-results.php===		===show-results.php===
	This script handles the analysis and display of the regression test data that resides in the SQL tables created by <code>post-results.php</code>. Per regression test, it operates with the folllowing parameters: (a) a table name; (b) an array of check columns, i.e., columns that contain the actual test data that we're concerned with; and (c) an array of control columns, i.e., columns whose values determine the grouping of data within each test table, e.g., if one's control columns are 'processor' and 'node_count', then <code>show-results.php</code> will discover every possible combination (between columns) of distinct column cell values and use that set of combinations to group test data prior to analysis and display. Each grouping combination includes exactly one value from every control column. This is easier understood when seen than when described (due in a large part to my inability to describe the system at hand), so visit the [http://cluster.earlham.edu/~~~tmcnulty~~/bccdrt/ actual script] and see what it does for yourself!		This script handles the analysis and display of the regression test data that resides in the SQL tables created by <code>post-results.php</code>. Per regression test, it operates with the folllowing parameters: (a) a table name; (b) an array of check columns, i.e., columns that contain the actual test data that we're concerned with; and (c) an array of control columns, i.e., columns whose values determine the grouping of data within each test table, e.g., if one's control columns are 'processor' and 'node_count', then <code>show-results.php</code> will discover every possible combination (between columns) of distinct column cell values and use that set of combinations to group test data prior to analysis and display. Each grouping combination includes exactly one value from every control column. This is easier understood when seen than when described (due in a large part to my inability to describe the system at hand), so visit the [http://cluster.earlham.edu/~tobias/bccdrt/ actual script] and see what it does for yourself!

	So, to make <code>show-results.php</code> aware of your newly-defined regression test, simply create an associative array containing your table name (<code>tbname</code>), array of check columns (<code>check_cols</code>), and array of control columns (<code>control_cols</code>) and append it to the <code>$tests</code> array. The table name is the test name prefixed by <code>rt_</code>, and the column names are exactly the same as those passed to <code>postresult(...)</code> in the original Python.		So, to make <code>show-results.php</code> aware of your newly-defined regression test, simply create an associative array containing your table name (<code>tbname</code>), array of check columns (<code>check_cols</code>), and array of control columns (<code>control_cols</code>) and append it to the <code>$tests</code> array. The table name is the test name prefixed by <code>rt_</code>, and the column names are exactly the same as those passed to <code>postresult(...)</code> in the original Python.

Tmcnulty: move from Powerpc

2005-10-07T03:16:43Z

move from Powerpc

New page

=Regression Testing=
==Executing the Tests==
To execute the regression tests included with bccdrt on a running copy of the BCCD, simply checkout the <code>bccdrt</code> module from CVS on <code>bccd.cs.uni.edu</code> and start <code>runtests.py</code>:
<pre>
export CVSROOT=username@bccd.cs.uni.edu:/var/lib/cvs/var/lib/cvs # you aren't seeing double and this isn't a typo!
cvs co bccdrt
cd bccdrt
./runtests.py # run list-packages and install python first if need be
</pre>

==System Overview==
This section exposes the architecture behind bccdrt, which refers generally to the collection of Python and PHP scripts than handle the execution and data extraction (<code>bccdrt.py</code>), recording (<code>post-results.php</code>), and analysis (<code>show-results.php</code>) of BCCD regression tests (the scripts were written and tested on PPC, but can be used on any architecture the BCCD supports).

===htdocs===
This directory houses the PHP scripts that record and analyze test data.

===rt.d===
This directory houses individual test scripts and the bccdrt library, which provides several routines to the individual tests (see [[#bccdrt.py|bccdrt.py]] below).

===runtests.py===
This script calls some generic bccd cluster initialization routines (<code>bccd-allowall</code>, <code>bccd-snarfhosts</code>, and <code>bccd-checkem machines</code>) and then executes each of the tests in the <code>rt.d</code> directory. No per-test modification of <code>runtests.py</code> is necessary.

===bccdrt.py===
This library provides several routines to individal tests, including <code>system(...)</code>, <code>runcmd(...)</code>, and <code>postresult(...)</code>. It resides in the <code>rt.d</code> directory, but contains no actual test code itself.

*<code>system(...)</code> takes a single argument in the form of a shell command, executes that command, and allows the output to print to the console.
*<code>runcmd(...)</code> does the same thing, but returns the output (both stdout and stderr) as a string.
*<code>postresult(...)</code> takes two arguments: the test name and an associative array of your test data (column) names and values. <code>postresult(...)</code> adds the generic test data, like machine name, BCCD release, etc., encodes the data in XML, and posts it to <code>post-results.php</code> via HTTP.

To define a new test, write a new script in Python that executes whatever programs on the BCCD the test involves and then extracts the relevant data from them (e.g., with regular expressions). There is an sample test in <code>rt.d</code> named <code>gromacs.py</code>. The only call your test function must make is to <code>postresult(...)</code>.

'''Example: Creating the associative array and calling <code>postresult(...)</code>'''
<pre>
import bccdrt

result={}

result["molecule"] = os.path.basename(moleculedir)
result["np"] = np
result["walltime"] = Real
result["psNODEhr"] = psNODEhr

postresult("gromacs",result)
</pre>

===post-results.php===
This script is generally responsible for recording data as posted by <code>bccdrt.py</code>. This involves the following:
* receiving the XML-encoded data from <code>bccdrt.py</code>
* determining the SQL table format from the given data names and values
* creating the SQL table if necessary
* inserting the given data into the table

'''What does this mean for you?''' <code>post-results.php</code> is entirely automatic; that is, it requires no per-test modification to operate.

===show-results.php===
This script handles the analysis and display of the regression test data that resides in the SQL tables created by <code>post-results.php</code>. Per regression test, it operates with the folllowing parameters: (a) a table name; (b) an array of check columns, i.e., columns that contain the actual test data that we're concerned with; and (c) an array of control columns, i.e., columns whose values determine the grouping of data within each test table, e.g., if one's control columns are 'processor' and 'node_count', then <code>show-results.php</code> will discover every possible combination (between columns) of distinct column cell values and use that set of combinations to group test data prior to analysis and display. Each grouping combination includes exactly one value from every control column. This is easier understood when seen than when described (due in a large part to my inability to describe the system at hand), so visit the [http://cluster.earlham.edu/~tmcnulty/bccdrt/ actual script] and see what it does for yourself!

So, to make <code>show-results.php</code> aware of your newly-defined regression test, simply create an associative array containing your table name (<code>tbname</code>), array of check columns (<code>check_cols</code>), and array of control columns (<code>control_cols</code>) and append it to the <code>$tests</code> array. The table name is the test name prefixed by <code>rt_</code>, and the column names are exactly the same as those passed to <code>postresult(...)</code> in the original Python.

'''Example: Defining a test in show-results.php'''
<pre>
$tests = array( array( "tbname" => "rt_gromacs",
"check_cols" => array("walltime","psNODEhr"),
"control_cols" => array("machine","molecule","np") )
);
</pre>

<code>show-results.php</code> then iterates through <code>$tests</code> in the body of the HTML document, calling <code>showResults(...)</code> with the parameters defined in each element.

==Recap: Creating a New Test==
That's a lot of talk for a relatively simple process. Here's the basic outline once more of what creating a new regression test involves:

* 1. write the actual test in Python and call <code>postresult(...)</code> to send the test data to the result server
* 2. drop your test in the <code>rt.d</code> directory
* 3. define your test's table name, check columns, and control columns in <code>show-results.php</code>
* 4. commit your changes to CVS, update the CVS module on the result server, and [[#Executing_the_Tests|execute the tests]].

==FAQ==
===How does show-results.php analyze test data?===
The analysis script calculates the standard deviation of each grouping of each check column and from that determines, on a per-cell basis, outliers in the data set, i.e., possible test failures (1*S). Cells that pass the test are given the CSS class <code>rt_pass</code>, and cells that fail <code>rt_fail</code>. Currently, the stylesheet marks passes green and potential failures red.

==TODO==
* Implement early failure detection and reporting (e.g., "this command doesn't even run")

==Conclusion==
That brings a close to our discussion of BCCD regression testing. Please direct any questions, comments, or suggestions to the system's author, [mailto:tmcnulty@ppckernel.org Tobias McNulty].

BCCD:Regression Testing - Revision history

Tmcnulty at 18:30, 30 March 2008

Tmcnulty: move from Powerpc