Checkpoint and Restarting - Revision history

Mccoyjo at 20:37, 6 July 2005

2005-07-06T20:37:15Z

← Older revision		Revision as of 20:37, 6 July 2005
Line 1:		Line 1:
			==Checkpointing and Restarting Methods in GROMACS==

			There are three checkpointing and restarting operations dealing with the GROMACS tools we care about for Folding@Clusters: creating an input file to start a run (our traditional grompp'ing), running the MD, and restarting a run from a failed run. The following shows the minimum set of files needed for each operation. Output files are also noted. The command line options are noted in []'s after each file.

			*creating an input file (using grompp)
			**Input
			***mdp [-f]
			***gro [-c]
			***top [-p]
			**Output
			***tpr [-o]
			***mdp [-po]

			*running the MD (mdrun)
			**Input
			***tpr [-s]
			**Output
			***trr [-o]
			***gro [-c]
			***log [-g]
			***edr [-e]

			*restart (grompp)
			**Input
			***tpr [-c]
			***trr [-t]
			***mdp [-f]
			***top [-p]
			**Output
			***tpr [-o]

			The set of files needed from the assignment server are the mdp, gro, and top files (molecule.conf should be in this set but it isn't relevant to this doc). The rest of the files needed can be generated from this set. Previous notions of running simulations with only a gro file or a top file are faulty; both filetypes represent different sets of critical information to MD. The gro file contains the name of atoms, their positions, and their velocities. The topology of a molecular system are contained in the top file and consists of information like bonds, pairs, angles, and dihedrals. The mdp file is essential in that it specifies what the parameters of the MD simulation. One should also note that this is just a barebones list of the minimum requirements of using these tools. Also to be noted is the fact that there are more command arguments in these lists than is necessary. If run without all the argments listed above, the tools will automatically assume the files are present in the working directory.

			Notes from the online GROMACS documentation suggest that the most accurate simulation restarts include an edr (energy) file. We may want to start including this in our restart methods.

	<h2>Checkpoint Frequency</h2>		<h2>Checkpoint Frequency</h2>

Mccoyjo at 20:51, 1 June 2005

2005-06-01T20:51:21Z

← Older revision		Revision as of 20:51, 1 June 2005
Line 24:		Line 24:
	<h2>Using GROMACS tools</h2>		<h2>Using GROMACS tools</h2>

	~~There are several cases to consider~~ when using the ~~GROMACS suite~~ to ~~checkpoint~~ and restart.		Notes:
			<ul>
			<li>Using only tpr files as input fails when using <tt>grompp</tt> to change the number of processes the tpr file is prepared for.</li>
			<li><tt>grompp</tt> requires a .top file to generate tpr files.</li>
			<li>Using both trr and edr files when restarting with <tt>grompp</tt> results in a more accurrate restart (see the <tt>grompp</tt> section in the GROMACS 3.2 online documentation).</li>
			</ul>


Line 53:		Line 57:
	<tt>trjconv -s original.tpr -f checkpoint.trr -o new.gro</tt></li>		<tt>trjconv -s original.tpr -f checkpoint.trr -o new.gro</tt></li>
	<li> Use the .gro as input to grompp to create a new .tpr file configured for the new number of nodes.<br />		<li> Use the .gro as input to grompp to create a new .tpr file configured for the new number of nodes.<br />
	<tt>grompp -f mdout.mdp -c new.gro -np 4 -o new.tpr</tt></li>		<tt>grompp -f mdout.mdp -c new.gro -p topol.top -np 4 -o new.tpr</tt></li>
	</ol>		</ol>
	Notes:<br />		Notes:<br />

Mccoyjo at 20:37, 1 June 2005

2005-06-01T20:37:42Z

← Older revision		Revision as of 20:37, 1 June 2005
Line 52:		Line 52:
	<li> Create a .gro file that incorporates the result and the topology files.<br />		<li> Create a .gro file that incorporates the result and the topology files.<br />
	<tt>trjconv -s original.tpr -f checkpoint.trr -o new.gro</tt></li>		<tt>trjconv -s original.tpr -f checkpoint.trr -o new.gro</tt></li>
	<li> Use the .gro as input to grompp to create a new .tpr file configured for the new number of nodes.</br>		<li> Use the .gro as input to grompp to create a new .tpr file configured for the new number of nodes.<br />
	<tt>grompp -f mdout.mdp -c new.gro -np 4 -o new.tpr</tt></li>		<tt>grompp -f mdout.mdp -c new.gro -np 4 -o new.tpr</tt></li>
	</ol>		</ol>

Mccoyjo at 20:37, 1 June 2005

2005-06-01T20:37:01Z

← Older revision		Revision as of 20:37, 1 June 2005
Line 57:		Line 57:
	Notes:<br />		Notes:<br />

	A simulation can only be ran for a set amount of time without modification. Even after restarting, the simulation cannot continue past the simulation time specified when grompp was initially executed. To extend the simulation, tpbconv must be used with the -until or -extend (which take a number of picoseconds as arguments) options:<br />		<p> A simulation can only be ran for a set amount of time without modification. Even after restarting, the simulation cannot continue past the simulation time specified when grompp was initially executed. To extend the simulation, tpbconv must be used with the -until or -extend (which take a number of picoseconds as arguments) options:</p>
	<tt>tpbconv -s topol.tpr -f traj.trr -e ener.edr -o new.tpr -extend 10</tt>		<tt>tpbconv -s topol.tpr -f traj.trr -e ener.edr -o new.tpr -extend 10</tt>

Mccoyjo at 20:35, 1 June 2005

2005-06-01T20:35:10Z

← Older revision		Revision as of 20:35, 1 June 2005
Line 21:		Line 21:

	Neither of these signals provide useful checkpointing mechanism. With some modicition (remove the code that modifies <tt>nsteps</tt>), the SIGUSR1 mechanism could be useful. Note that we are already using SIGINT in nanny/child communication and SIGINT is our only free signal due to COSM.		Neither of these signals provide useful checkpointing mechanism. With some modicition (remove the code that modifies <tt>nsteps</tt>), the SIGUSR1 mechanism could be useful. Note that we are already using SIGINT in nanny/child communication and SIGINT is our only free signal due to COSM.

			<h2>Using GROMACS tools</h2>

			There are several cases to consider when using the GROMACS suite to checkpoint and restart.



			Altnernative method:

			tools used:
			<ul>
			<li>trjconv</li>
			<li>grompp</li>
			</ul>
			files initially required:
			<ul>
			<li> original mdout.mdp </li>
			<li> original .tpr </li>
			<li> original mdout.mdp</li>
			<li> result or checkpoint file (.trr)</li>
			<li> topology file (.top)</li>
			</ul>
			files created:
			<ul>
			<li>new .gro</li>
			<li>new .tpr</li>
			</ul>
			Process:
			<ol>
			<li> Create a .gro file that incorporates the result and the topology files.<br />
			<tt>trjconv -s original.tpr -f checkpoint.trr -o new.gro</tt></li>
			<li> Use the .gro as input to grompp to create a new .tpr file configured for the new number of nodes.</br>
			<tt>grompp -f mdout.mdp -c new.gro -np 4 -o new.tpr</tt></li>
			</ol>
			Notes:<br />

			A simulation can only be ran for a set amount of time without modification. Even after restarting, the simulation cannot continue past the simulation time specified when grompp was initially executed. To extend the simulation, tpbconv must be used with the -until or -extend (which take a number of picoseconds as arguments) options:<br />
			<tt>tpbconv -s topol.tpr -f traj.trr -e ener.edr -o new.tpr -extend 10</tt>

Mccoyjo at 20:18, 1 June 2005

2005-06-01T20:18:44Z

← Older revision		Revision as of 20:18, 1 June 2005
Line 6:		Line 6:
	</li></ol>		</li></ol>

	<h2>~~Signals to~~ <tt>mdrun</tt> and their relevance to the checkpoint/restart process</h2>		<h2>Signal accepted by <tt>mdrun</tt> and their relevance to the checkpoint/restart process</h2>

	The <tt>mdrun</tt> process accepts SIGTERM and SIGUSR1. These signals can be received by a <tt>mdrun</tt> process of any rank. The effects of the signals are as follows:		The <tt>mdrun</tt> process accepts SIGTERM and SIGUSR1. These signals can be received by a <tt>mdrun</tt> process of any rank. The effects of the signals are as follows:
Line 19:		Line 19:
	</ul></li>		</ul></li>
	</ul>		</ul>

			Neither of these signals provide useful checkpointing mechanism. With some modicition (remove the code that modifies <tt>nsteps</tt>), the SIGUSR1 mechanism could be useful. Note that we are already using SIGINT in nanny/child communication and SIGINT is our only free signal due to COSM.

Mccoyjo at 20:15, 1 June 2005

2005-06-01T20:15:30Z

← Older revision		Revision as of 20:15, 1 June 2005
Line 12:		Line 12:
	<li><b>SIGTERM</b>		<li><b>SIGTERM</b>
	<ul>		<ul>
	<li>Sets nsteps to current steps plus one.</li>		<li>Sets <tt>nsteps</tt> to current steps plus one.</li>
	</ul></li>		</ul></li>

	<li><b>SIGUSR1</b><ul>		<li><b>SIGUSR1</b><ul>
	<li>Sets nsteps the next multiple of nstxout past the current step.</li>		<li>Sets <tt>nsteps</tt> the next multiple of <tt>nstxout</tt> past the current step.</li>
	</ul></li>		</ul></li>
	</ul>		</ul>

Mccoyjo at 20:14, 1 June 2005

2005-06-01T20:14:31Z

← Older revision		Revision as of 20:14, 1 June 2005
Line 12:		Line 12:
	<li><b>SIGTERM</b>		<li><b>SIGTERM</b>
	<ul>		<ul>
	<li>Sets nsteps to current steps ~~plu~~ one.</li>		<li>Sets nsteps to current steps plus one.</li>
	</ul></li>		</ul></li>

	<li>~~SIGUSR1~~<ul>		<li><b>SIGUSR1</b><ul>
	<li><~~/li~~>		<li>Sets nsteps the next multiple of nstxout past the current step.</li>
	<~~/ul~~>Sets nsteps the next multiple of nstxout past the current step.</li>		</ul></li>
	</ul>		</ul>

Mccoyjo at 20:13, 1 June 2005

2005-06-01T20:13:45Z

← Older revision		Revision as of 20:13, 1 June 2005
Line 5:		Line 5:
	<li>The nanny checks the size of the checkpoint file. If the file has become larger, it is transferred to the mother. This check happens about every two seconds (as of 1 June 2005).		<li>The nanny checks the size of the checkpoint file. If the file has become larger, it is transferred to the mother. This check happens about every two seconds (as of 1 June 2005).
	</li></ol>		</li></ol>

			<h2>Signals to <tt>mdrun</tt> and their relevance to the checkpoint/restart process</h2>

			The <tt>mdrun</tt> process accepts SIGTERM and SIGUSR1. These signals can be received by a <tt>mdrun</tt> process of any rank. The effects of the signals are as follows:
			<ul>
			<li><b>SIGTERM</b>
			<ul>
			<li>Sets nsteps to current steps plu one.</li>
			</ul></li>

			<li>SIGUSR1<ul>
			<li></li>
			</ul>Sets nsteps the next multiple of nstxout past the current step.</li>
			</ul>

Mccoyjo at 20:07, 1 June 2005

2005-06-01T20:07:32Z

New page

<h2>Checkpoint Frequency</h2>

Checkpointing in Folding@Clusters has two distinct parts:
<ol><li>mdrun generating checkpoints in an interval given by the <tt>nstxout</tt> parameter in <tt>grompp.mdp</tt>.</li>
<li>The nanny checks the size of the checkpoint file. If the file has become larger, it is transferred to the mother. This check happens about every two seconds (as of 1 June 2005).
</li></ol>