CS382:Scraper - Revision history

Edlefma at 00:18, 16 April 2009

2009-04-16T00:18:52Z

← Older revision		Revision as of 00:18, 16 April 2009
Line 1:		Line 1:
	= Documentation =		= Documentation =
	==NAME==		==NAME==
	mwscraper - MediaWiki scraper tool builds dynamic MediaWiki pages from template files that scrape data from other pages.		mwscraper - MediaWiki scraper tool builds dynamic MediaWiki pages from template files that scrape data from other pages.

Edlefma at 17:02, 8 April 2009

2009-04-08T17:02:38Z

← Older revision		Revision as of 17:02, 8 April 2009
Line 1:		Line 1:
	= Documentation =		= Documentation =
	==NAME==		==NAME==
	mwscraper - MediaWiki scraper tool builds dynamic MediaWiki pages from template files that scrape data from other pages.		mwscraper - MediaWiki scraper tool builds dynamic MediaWiki pages from template files that scrape data from other pages.

Line 24:		Line 24:
	; Crypt::SSLeay		; Crypt::SSLeay
	:Allows https URI's for the wiki		:Allows https URI's for the wiki
			; Compress::Zlib
			:Needed to decompress responses from the wiki.

	==OPTIONS==		==OPTIONS==

Edlefma at 02:48, 8 April 2009

2009-04-08T02:48:27Z

← Older revision		Revision as of 02:48, 8 April 2009
Line 52:		Line 52:
	The template format is that of the Template perl module. See the Template Perldoc page for more details.		The template format is that of the Template perl module. See the Template Perldoc page for more details.

	Template code is placed between [% %] blocks and allows a wide range of functionality. The most basic form is inserting the value of a variable. The form of this is simply ~~[% variable_name %]~~		Template code is placed between [% %] blocks and allows a wide range of functionality. The most basic form is inserting the value of a variable. The form of this is simply

	You can also assign a value to variable using the = operator. ~~[% variable_name = expression %]~~		<code>
			[% variable_name %]
			</code>
			You can also assign a value to variable using the = operator.

			<code>
			[% variable_name = expression %]
			</code>
	===EXAMPLE===		===EXAMPLE===
	software.tt:		software.tt:

Edlefma at 02:47, 8 April 2009

2009-04-08T02:47:15Z

← Older revision		Revision as of 02:47, 8 April 2009
Line 4:		Line 4:

	==SYNOPSIS==		==SYNOPSIS==
	mwscraper --help\|--usage		mwscraper --help\|--usage\|--version

	mwscraper [--host URL] [--login] [--username USERNAME] ] [--password PASSWORD] [--upload] ~~TEMPLATE~~		mwscraper [--host URL] [--login] [--username USERNAME] ] [--password PASSWORD] [--upload] TEMPLATES...

	==DESCRIPTION==		==DESCRIPTION==
Line 36:		Line 36:
	; -e\|--edit\|--upload		; -e\|--edit\|--upload
	:Tells the scraper to upload the generated pages to the wiki. The name of the page it will be prompted for unless the template specifies one using the title function. If the page does not already exist a new one will be created.		:Tells the scraper to upload the generated pages to the wiki. The name of the page it will be prompted for unless the template specifies one using the title function. If the page does not already exist a new one will be created.
	; --help, --usage		; --help, --usage, --version
	:Displays help/usage information.		:Displays help/usage/version information.

	==ENVIRONMENT==		==ENVIRONMENT==

Edlefma at 02:35, 8 April 2009

2009-04-08T02:35:02Z

Show changes

Nate: /* DESCRIPTION */

2009-04-08T00:53:29Z

DESCRIPTION

← Older revision		Revision as of 00:53, 8 April 2009
Line 23:		Line 23:
	You can pass it a host url for the wiki, or by default it uses wiki.cs.earlham.edu. If no username or password is provided they will be prompted. By default, the generated page will be printed to stdout but if the --upload flag is provided the page will be directly uploaded to the wiki using the title specified by the title function. If the template does not specify a title for the generated page, one will be prompted for.		You can pass it a host url for the wiki, or by default it uses wiki.cs.earlham.edu. If no username or password is provided they will be prompted. By default, the generated page will be printed to stdout but if the --upload flag is provided the page will be directly uploaded to the wiki using the title specified by the title function. If the template does not specify a title for the generated page, one will be prompted for.

	~~See the~~ [[CS382:Scraper_Recipes]]
			You will want to look at [[CS382:Scraper_Recipes]] to get the most out of this tool.

	==DEPENDENCIES==		==DEPENDENCIES==

Nate at 00:52, 8 April 2009

2009-04-08T00:52:47Z

← Older revision		Revision as of 00:52, 8 April 2009
Line 22:		Line 22:

	You can pass it a host url for the wiki, or by default it uses wiki.cs.earlham.edu. If no username or password is provided they will be prompted. By default, the generated page will be printed to stdout but if the --upload flag is provided the page will be directly uploaded to the wiki using the title specified by the title function. If the template does not specify a title for the generated page, one will be prompted for.		You can pass it a host url for the wiki, or by default it uses wiki.cs.earlham.edu. If no username or password is provided they will be prompted. By default, the generated page will be printed to stdout but if the --upload flag is provided the page will be directly uploaded to the wiki using the title specified by the title function. If the template does not specify a title for the generated page, one will be prompted for.

			See the [[CS382:Scraper_Recipes]]

	==DEPENDENCIES==		==DEPENDENCIES==

Edlefma at 14:55, 6 April 2009

2009-04-06T14:55:35Z

← Older revision		Revision as of 14:55, 6 April 2009
Line 1:		Line 1:
	= Documentation =		= Documentation =
	==NAME==		==NAME==
	mwscraper - MediaWiki scraper tool builds dynamic MediaWiki pages from template files that scrape data from other pages.		mwscraper - MediaWiki scraper tool builds dynamic MediaWiki pages from template files that scrape data from other pages.

	==SYNOPSIS==		==SYNOPSIS==
	mwscraper [--host URL] [--~~user~~ USERNAME] [--password PASSWORD] [--upload] TEMPLATES...		mwscraper --help

			mwscraper [--host URL] [--login [USERNAME] ] [--password PASSWORD] [--upload] TEMPLATES...

			==OPTIONS==
			; -h\|--host
			:Sets the url of the wiki to use. Should start with either http:// or https://.
			; -l\|--login
			:Tells the scraper to attempt to login if it needs to access the wiki (either for reading or uploading). If a username is not given it will be prompted for. If a template uses the login function, it will override this option.
			; -p\|--password
			:Provides a password to be used when logging in. This option only works if --login was specified with a username.
			; -u\|--upload
			:Tells the scraper to upload the generated pages to the wiki. The name of the page it will be prompted for unless the template specifies one using the title function.

	==DESCRIPTION==		==DESCRIPTION==
Line 15:		Line 27:
	* MediaWiki::API		* MediaWiki::API
	* Template		* Template
			* Crypt::SSLeay (for https)

	==TEMPLATES==		==TEMPLATES==
	The template format is that of the Template perl module. Here is an example:		The template format is that of the Template perl module. Here is an example:

	software.tt:		software.tt:
	[% title('CS382:Software') -%]		[% title('CS382:Software') -%]
	= Software =		= Software =
Line 43:		Line 57:
	; prompt( STRING )		; prompt( STRING )
	:Prompts the user in the form 'STRING: ' and then returns the next entered line without the trailing newline.		:Prompts the user in the form 'STRING: ' and then returns the next entered line without the trailing newline.
			; login( [USERNAME [, PASSWORD ] ] )
			:Logs in to the wiki using USERNAME and PASSWORD. If one or both aren't provided they will be prompted for.
	; cat( STRING... )		; cat( STRING... )
	:Concatenates all passed strings together and returns the result.		:Concatenates all passed strings together and returns the result.

Edlefma at 22:49, 31 March 2009

2009-03-31T22:49:01Z

← Older revision		Revision as of 22:49, 31 March 2009
Line 181:		Line 181:
	[% INCLUDE GenEd name = 'Emphasizes and provides first-hand experience with both theoretical analysis and the collection of empirical data.' anchor = '#Scientific Inquiry Requirement' %]		[% INCLUDE GenEd name = 'Emphasizes and provides first-hand experience with both theoretical analysis and the collection of empirical data.' anchor = '#Scientific Inquiry Requirement' %]

	~~</pre>~~

	~~== scraper.tt ==~~
	~~<pre>~~
	~~[% title('CS382:Scraper') ~%]~~

	~~[% BLOCK Example -%]~~
	~~== [% file %] ==~~
	~~<pre>~~
	~~[% INSERT $file -%]~~
	~~</pre>~~
	~~[% END ~%]~~

	~~= Documentation =~~
	~~[% INSERT mwscraper.wiki ~%]~~

	~~= Examples =~~
	~~[% INCLUDE Example file = 'software.tt' %]~~
	~~[% INCLUDE Example file = 'geneds.tt' %]~~
	~~[% INCLUDE Example file = 'scraper.tt' %]~~
	</pre>		</pre>

Edlefma at 22:48, 31 March 2009

2009-03-31T22:48:17Z

← Older revision		Revision as of 22:48, 31 March 2009
Line 181:		Line 181:
	[% INCLUDE GenEd name = 'Emphasizes and provides first-hand experience with both theoretical analysis and the collection of empirical data.' anchor = '#Scientific Inquiry Requirement' %]		[% INCLUDE GenEd name = 'Emphasizes and provides first-hand experience with both theoretical analysis and the collection of empirical data.' anchor = '#Scientific Inquiry Requirement' %]

			</pre>

			== scraper.tt ==
			<pre>
			[% title('CS382:Scraper') ~%]

			[% BLOCK Example -%]
			== [% file %] ==
			<pre>
			[% INSERT $file -%]
			</pre>
			[% END ~%]

			= Documentation =
			[% INSERT mwscraper.wiki ~%]

			= Examples =
			[% INCLUDE Example file = 'software.tt' %]
			[% INCLUDE Example file = 'geneds.tt' %]
			[% INCLUDE Example file = 'scraper.tt' %]
	</pre>		</pre>