SE2006:group bar:scrapi - Revision history

Hunteke: making obsolete

2006-03-31T15:41:33Z

making obsolete

@@ Line 1: / Line 1: @@
 ==Defining Sources==
 Begin by starting at the bottom of the scrape. Create a regular expression.
@@ Line 24: / Line 52: @@
 Here you can see how the Source will pick off @@@theater_id@@@ variable and pass it into its subSource, theaterInfo. Whatever variables appear in the subSource's URL will be expanded based on variable values it inherits from its parent.

Lemanal at 20:21, 30 March 2006

2006-03-30T20:21:20Z

← Older revision		Revision as of 20:21, 30 March 2006
Line 31:		Line 31:
	/* add to me */		/* add to me */
	private void createTypes() {		private void createTypes() {
	addType("href_tag","text","(<A href.*?>)");		addType("href_tag","text","(<A href>[\\\\s\\\\S]*?)");
	addType("string","text","([\\\\s\\\\S]*?)");		addType("string","text","([\\\\s\\\\S]*?)");
	addType("int","integer","(\\d*)");		addType("int","integer","(\\d*)");

Lemanal at 20:19, 30 March 2006

2006-03-30T20:19:58Z

← Older revision		Revision as of 20:19, 30 March 2006
Line 6:		Line 6:
	</pre>		</pre>

	Note that here we use variable names and variable types. The variable names are the eventual keys in the hash returned by the scrape. These names will be replaced by a regular expression associated with the type. The variable name is also associated with a SQL type.		Note that here we use variable names and variable types. The variable names are the eventual keys in the hash returned by the scrape and thus the eventual column names in the database. These names will be replaced by a regular expression associated with the type. The variable name is also associated with a SQL type.

	Next, based on this regular expression and a URL create a Source class.		Next, based on this regular expression and a URL create a Source class.
Line 14:		Line 14:
	</pre>		</pre>

	Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and ~~expanded based on~~ the ~~regExp associated with the that variable's type~~. The parent Source will create a new Source for each regexp match it makes on its URL. It is defined with its subSource as an extra argument to the constructor.		Note that here we have a reference to @@@theater_id@@@ in the URL. This value will inherited from the parent source and filled in the URL. The parent Source will create a new Source for each regexp match it makes on its URL. It is defined with its subSource as an extra argument to the constructor.

	<pre>		<pre>
Line 31:		Line 31:
	/* add to me */		/* add to me */
	private void createTypes() {		private void createTypes() {
	addType("href_tag","text","(<A href.*?>)");		addType("href_tag","text","(<A href.*?>)");
	addType("string","~~string~~","(.*?)");		addType("string","text","([\\\\s\\\\S]*?)");
	addType("int","integer","(\\d*)");		addType("int","integer","(\\d*)");
	addType("state_abrev", "~~string~~","(\\\\w\\\\w)");		addType("state_abrev", "varchar(2)","(\\\\w\\\\w)");
	addType("theater_num","integer","([^0].*?)");		addType("state_abrev", "varchar(2)","(\\\\w\\\\w)");
			addType("state_census", "varchar(9)", "(04000US\\\\d\\\\d)");
			addType("theater_num","integer","([^0].*?)");
			addType("two_char","varchar(2)","(\\\\w\\\\w)");
			addType("one_char","varchar(1)","(\\\\w)");
			addType("float","float","([-+]?[0-9]*\\.?[0-9]+)");
			addType("big_int","int","([0-9,]*?)");
	}		}

	private void addType(String typeName,String sqlType, String RegExp) { .. }		private void addType(String typeName,String sqlType, String RegExp) { .. }
	</pre>		</pre>

	==Skipping Groups==		==Skipping Groups==
	Note that ~~currently the only way~~ to ~~have~~ a group in ~~your~~ regular expression ~~without messing up our framework is~~ to ~~have it matched as~~ a variable. ~~This means~~ that you will ~~have to define~~ a ~~type~~ for the group you want and use that in your regular expression. We have been using the skip variable name which we would not pass on into the database. If you have more than one skip variable of different types you will need to give them different names ie skip_n.		Note that every type when expanded to a regExp becomes a group. We use the ordering of these groups in the regular expression to make a back reference number to variable name mapping. In order to ensure that any groups you add to a regular expression outside of what will be expanded by a variable will not be picked up as a value for a variable you must make sure that the group is not a back reference. Put a '?' as the first character in that group, e.g., (?.+)for the group you want and use that is in your regular expression.

Lemanal at 21:52, 17 March 2006

2006-03-17T21:52:17Z

← Older revision		Revision as of 21:52, 17 March 2006
Line 23:		Line 23:
	</pre>		</pre>

	Here you can see how the Source will pick off @@@theater_id@@@ variable and pass it into its subSource, theaterInfo.		Here you can see how the Source will pick off @@@theater_id@@@ variable and pass it into its subSource, theaterInfo. Whatever variables appear in the subSource's URL will be expanded based on variable values it inherits from its parent.

	==Adding New Types==		==Adding New Types==

Lemanal at 21:49, 17 March 2006

2006-03-17T21:49:20Z

← Older revision		Revision as of 21:49, 17 March 2006
Line 14:		Line 14:
	</pre>		</pre>

	Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and expanded based on the regExp associated with the that variable's type. The parent Source will create a new Source for each regexp match it makes its on URL. It is defined with its subSource as an extra argument to the constructor.		Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and expanded based on the regExp associated with the that variable's type. The parent Source will create a new Source for each regexp match it makes on its URL. It is defined with its subSource as an extra argument to the constructor.

	<pre>		<pre>

Lemanal at 21:49, 17 March 2006

2006-03-17T21:49:06Z

← Older revision		Revision as of 21:49, 17 March 2006
Line 14:		Line 14:
	</pre>		</pre>

	Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and expanded based on the regExp associated with the that variable's type. The parent Source will create a new Source for each regexp match it makes its URL. It is defined with its subSource as an extra argument to the constructor.		Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and expanded based on the regExp associated with the that variable's type. The parent Source will create a new Source for each regexp match it makes its on URL. It is defined with its subSource as an extra argument to the constructor.

	<pre>		<pre>

Lemanal at 21:26, 17 March 2006

2006-03-17T21:26:04Z

← Older revision		Revision as of 21:26, 17 March 2006
Line 14:		Line 14:
	</pre>		</pre>

	Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and expanded. The parent Source will create a new Source for each regexp match it makes its URL. It is defined with its subSource as an extra argument to the constructor.		Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and expanded based on the regExp associated with the that variable's type. The parent Source will create a new Source for each regexp match it makes its URL. It is defined with its subSource as an extra argument to the constructor.

	<pre>		<pre>

Lemanal at 21:24, 17 March 2006

2006-03-17T21:24:55Z

← Older revision		Revision as of 21:24, 17 March 2006
Line 6:		Line 6:
	</pre>		</pre>

	Note that here we use variable names and variable types. The variable names are the eventual ~~output from~~ the hash returned by the scrape. These names will be replaced by a regular expression associated with the type. The variable name is also associated with a SQL type.		Note that here we use variable names and variable types. The variable names are the eventual keys in the hash returned by the scrape. These names will be replaced by a regular expression associated with the type. The variable name is also associated with a SQL type.

	Next, based on this regular expression and a URL create a Source class.		Next, based on this regular expression and a URL create a Source class.

Lemanal at 21:24, 17 March 2006

2006-03-17T21:24:24Z

← Older revision		Revision as of 21:24, 17 March 2006
Line 6:		Line 6:
	</pre>		</pre>

	Note that here we use variable names and variable types. The variable names are the eventual output from the hash. These names will be replaced by a regular expression associated with the type. The variable name is also associated with a SQL type.		Note that here we use variable names and variable types. The variable names are the eventual output from the hash returned by the scrape. These names will be replaced by a regular expression associated with the type. The variable name is also associated with a SQL type.

	Next, based on this regular expression and a URL create a Source class.		Next, based on this regular expression and a URL create a Source class.

Lemanal at 21:23, 17 March 2006

2006-03-17T21:23:56Z

← Older revision		Revision as of 21:23, 17 March 2006
Line 14:		Line 14:
	</pre>		</pre>

	Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and expanded. The parent Source will create a new Source for each regexp match it makes its URL. It is defined with ~~it's~~ subSource as an extra argument to the constructor.		Note that here we have a reference to @@@theater_id@@@ in the URL. This will inherited from the parent source and expanded. The parent Source will create a new Source for each regexp match it makes its URL. It is defined with its subSource as an extra argument to the constructor.

	<pre>		<pre>