Commit 2bc3ed39 authored by Romain Bignon's avatar Romain Bignon

include thirdparty shotfactory (from browsershots)

parent f9988d96
......@@ -25,7 +25,7 @@ class Match(object):
self.data = None
class Browser(mechanize.Browser):
QUEUEDIR = '/var/lib/browsershot/queue/'
QUEUEDIR = 'queue'
FIRST_STR = '<b>GOATSE</b>'
IMG_URL = 'http://parano.me/h'
FINAL_STR = '<img src=%s />' % IMG_URL
......
#!/bin/sh
if [ ! -d screenshots ]; then
mkdir screenshots
fi
if [ ! -d queue ]; then
mkdir queue
fi
cd thirdparty/shotfactory
./shotfactory.py -w 20 --queue ../../queue --output ../../screenshots
= Authors =
Browsershots was invented and written by Johann C. Rocholl.
== Contributors ==
Thanks for patches go to the following contributors, listed alphabetically.
* Andreas Schönfelder
* Marcin Król
* Nicko van Someren
If you want to be mentioned here, please [mailto:johann@browsershots.org
send email].
\ No newline at end of file
This diff is collapsed.
= Frequently Asked Questions =
[[PageOutline(2-3,,inline)]]
----
== Using the service ==
=== May I use the screenshots of my own webpage for any purpose? ===
Yes, even for inclusion in closed, for-cost works. It would be nice if
you'd mention that you used browsershots.org for the screenshots, but you
don't have to.
=== Will this service remain free of charge? ===
Yes. For the foreseeable future, there will be free screenshots for
everybody. So far, there are no fixed plans for a commercial model. If and
when a paid priority service is introduced, the free service will continue.
=== How long will my screenshots stay on the server? ===
Screenshots will be deleted when they haven't been accessed in ten days.
This means that if you post a link to your screenshots on your blog, and
people look at them again and again, the screenshots will stay on the
server forever. If you want to have your screenshots removed manually, send
email to [mailto:johann@browsershots.org johann@browsershots.org].
=== Why are all browsers selected by default? ===
You're right that it would probably reduce the queue length if users would
have to select each browser that they want to test. But the main goal of
this project is to raise awareness for browser diversity. So I think it's
better to have all browsers enabled by default and find other ways to
reduce the length of the queue, like adding more screenshot factories. See
HowToCreateNewShotFactory if you want to help.
----
== Troubleshooting ==
=== Why don't I get any screenshots? ===
If the queue is very long, you may want to increase the '''Maximum Wait'''
option for your screenshot requests to prevent timeouts. The default is 30
minutes, but you can select up to 4 hours. Also, if your requested
configuration is too specific (e.g. specific Flash or Java version) there
may not be matching screenshot factories all the time. Try the '''Don't
care''' setting for quicker results.
=== Where can I find my screenshots? ===
Your screenshots will appear near the top on the same page where you select
the browsers. It will look like
[http://browsershots.org/website/http://browsershots.org/ this]. You can
bookmark that page to come back to it later to see your screenshots, and
also to request new screenshots of the same website. If you don't get any
screenshots on that page, please see the question above.
=== Some of my screenshots show a browser with a blank page. How come? ===
Our Linux browsers have 45 seconds to load and display a page. Mac and
Windows browsers get only 30 seconds because they don't run in parallel. If
this isn't enough time to display your page, your web server is probably
too slow or overloaded.
=== My menu appears twice in the screenshots. Why? ===
That's not a problem with your page but a side effect of the multi-page
merging. The screenshots are taken one screen at a time, and then merged
together to produce a tall screenshot. If your menu doesn't scroll with the
rest of the page, it will appear multiple times.
=== My screenshots are only the size of one screen, but my page is longer.
===
Sometimes the automatic scrolling doesn't work. The reason could be that
your page uses frames, or that it sets the keyboard focus to an input field
with JavaScript.
----
== Running a screenshot factory ==
=== I don't have Python on my computer. Can I still run a ShotFactory? ===
Python is included with recent versions of Linux and Mac OS, and on Windows
it should be easy to install.
If you really can't get Python to run on your computer, you could write
your own screenshot factory in your favorite programming language,
according to the FactoryInterface specification.
=== Where can I get a username/password for running a ShotFactory? ===
See HowToCreateNewShotFactory and send email to
[mailto:johann@browsershots.org johann@browsershots.org]. I'm going to add
an automatic web-based registration form "real soon now".
=== Can I use my computer while the ShotFactory is running? ===
On Mac and Windows, the screenshot factory program uses the desktop, so you
will have to stop it when you want to use the computer.
On Linux, the program runs in the background and uses a VNC server for the
screen. You can use your computer at the same time, and if the system load
is over a configurable limit, the screenshot factory program will stop
making screenshots until the system is idle again. The default load limit
is 1.0, but you can change it with the command line option -l.
=== xmlrpclib.!ProtocolError: <!ProtocolError for browsershots.org/RPC2:
404 Not Found> ===
That's because the /RPC2 interface was for the old version 0.3 of
Browsershots which moved to http://v03.browsershots.org/ for historic
reference. The current version 0.4 uses http://api.browsershots.org/xmlrpc/
for the screenshot factory API. You should install a package with "0.4" in
the name, or check out the [http://svn.browsershots.org/trunk/shotfactory/
trunk] from Subversion.
=== !RuntimeError: unsafe command '/usr/bin/opera -newpage' ===
For security reasons, the browser command must not contain whitespace or
start with a slash. The purpose of this mechanism is to avoid commands like
'''rm -rf /''' or '''/tmp/rootkit'''. The best solution is to create a
simple shell script with your desired options, make it executable (chmod
a+x), put it somewhere on your PATH and then use the name of the shell
script as the browser command.
----
== Background info ==
=== How did you come up with the idea? ===
I was inspired by the following existing services. One night in November
2004, I went to bed too early, and while I was lying awake, suddenly the
distributed volunteer idea struck me.
* [http://www.danvine.com/icapture/ Daniel Vine's iCapture]
* [http://fundisom.com/g5/ lixlpixel Safari Screenshot Generator]
* [http://www.browsercam.com/ BrowserCam]
=== How long has Browsershots been around? ===
Public beta test started in February 2005
([http://v02.browsershots.org/blog/2005/02/ old blog entry]). The first
public release (milestone:0.1.0) was in March 2005
([http://freshmeat.net/projects/browsershots/?branch_id=56805&release_id=189
981 article on freshmeat]). See the
[http://trac.browsershots.org/roadmap?show=all roadmap] for a release
history. The project was rewritten completely between milestone:0.2.6 and
milestone:0.3-alpha1.
=== Where do you get all those different machines? ===
This is the one feature that sets this project apart from similar online
services. The screenshots are made on distributed computers that are run by
volunteers. You can see a list of [http://browsershots.org/factories/
active screenshot factories]. Most of these are normal desktop machines
with domestic broadband internet. Distributed community processing ensures
O/S and browser diversity, but it makes quality assurance much harder. See
also DistributedArchitecture and FactoryInterface.
\ No newline at end of file
= Screenshot factory installation instructions =
Specific instructions for all supported platforms:
* INSTALL.LINUX
* INSTALL.WIN
* INSTALL.MAC
== Register your factory ==
* If you don't have a password yet, please
[https://browsershots.org/accounts/email/ create a user account].
* If you get an error saying '''Factory not found with name=...''', please
[https://browsershots.org/factories/add/ register your screenshot factory].
* If you get '''No browsers registered for factory ...''', please visit
[https://browsershots.org/browsers/add/ this page] with each of your
browsers.
== Troubleshooting ==
* Run '''shotfactory.py -h''' (or '''--help''') to see a list of command
line options.
* Run '''shotfactory.py -v''' (or '''--verbose''') to see more output,
including error messages from helper programs.
* On Linux, you can run '''xvncviewer :1''' to see the virtual screen of
the VNC server while a screenshot request is processed.
* You can also look at the files '''pgdn01.ppm''' or '''browsershot.png'''
to see if there's a problem.
* The file '''shotfactory.log''' contains error messages for failed
screenshot requests.
* If you use '''screen -L''', the file '''screenlog.0''' will contain the
full output of the shotfactory script.
* On Windows, if you run '''shotfactory.exe''' directly with a double
click, the terminal window may be closed immediately when an error occurs.
Try to start the terminal first and then run the program from the command
line to avoid this issue.
\ No newline at end of file
= Installing the screenshot factory on Linux =
This page has been updated to work with Browsershots 0.4.
== Install prerequisites ==
{{{
sudo apt-get install tightvncserver netpbm xautomation scrot subversion
}}}
== Get the screenshot factory source code ==
{{{
svn checkout http://svn.browsershots.org/trunk/shotfactory/
}}}
== Adjust config files ==
Change the file '''~/.vnc/xstartup''' to the following three lines. This
will set a white background and then run nothing except the window manager.
Be sure to remove any reference to xterm because it gets in the way.
{{{
#!/bin/sh
xsetroot -solid "#FFFFFF"
x-window-manager &
}}}
== Run ==
{{{
cd shotfactory
screen -L python shotfactory.py
}}}
See INSTALL for registration and troubleshooting tips.
\ No newline at end of file
= Installing the screenshot factory on Mac =
== Install prerequisites ==
* [http://www.codingmonkeys.de/mbo/ Subversion Binaries]
(or '''sudo port install subversion''')
* [http://netpbm.sourceforge.net/ NetPBM tools]
(or '''sudo port install netpbm''', or
[http://sourceforge.net/project/showfiles.php?group_id=7130&package_id=14464
binaries from Gallery])
* [http://www.versiontracker.com/dyn/moreinfo/macosx/19753 cscreen]
to change screen resolution
([http://download.browsershots.org/thirdparty/cscreen.dmg mirror])
* [http://appscript.sourceforge.net/download.html Python appscript module]
And if you want to support more media formats:
* Windows Media Player
[http://www.apple.com/downloads/macosx/video/windowsmediaplayerformacosx.htm
l for Mac OS X]
([http://www.microsoft.com/mac/otherproducts/otherproducts.aspx?pid=windowsm
edia alternate link, also for Mac OS 8.1-9])
== Get the source code ==
Check out the source code from the Subversion repository:
{{{
svn checkout http://svn.browsershots.org/trunk/shotfactory
}}}
Also copy the cscreen program into the '''shotfactory''' directory.
Starting with milestone:0.3-beta1, global installation with '''python
setup.py install''' is no longer required.
== Run ==
{{{
cd shotfactory
python shotfactory.py
}}}
See INSTALL for registration and troubleshooting tips.
\ No newline at end of file
= Installing ShotFactory on Windows =
'''Important security notice:''' don't run the screenshot factory on your
Administrator account. The program will load many websites in your browser,
and some of them may contain exploits for known security holes.
[BlogWindowsXpFactoryTakenOverBySpyware One of my screenshot factories has
been hijacked in this way.] It's safest to create a new unprivileged user
account and run the screenshot factory there.
See also StandaloneInternetExplorer if you want to run different versions
of Internet Explorer on one machine.
There are two different ways to install the screenshot factory on windows:
[[PageOutline(2,,inline)]]
== Simple method: binaries for Windows ==
This method is strongly recommended. It does not require any other
installations. Helper programs and the Python interpreter with all required
libraries are included.
* Download the latest zip file from the
[http://browsershots.org/downloads/binaries/ binaries download section]
* Extract the zip file somewhere (e.g. to a folder on the Desktop)
* Double click on '''shotfactory.exe''', or run it from the command line
See INSTALL for registration and troubleshooting tips.
== Advanced method: source code from the Subversion repository ==
Use this method if you want to fiddle with the source code.
* Install the following prerequisites:
* [http://subversion.tigris.org/project_packages.html#windows Subversion]
* [http://www.python.org/ Python]
* [http://sourceforge.net/projects/pywin32/ Python Win32 API]
* [http://www.pythonware.com/products/pil/ Python Imaging Library]
* [http://www.12noon.com/reschange.htm Resolution Changer (Console
version): reschangecon.exe]
* [http://www.teamcti.com/pview/prcview.htm Process Viewer (Command line
version): pv.exe]
* Check out the latest screenshot factory source code from
http://svn.browsershots.org/trunk/shotfactory/
* Starting with milestone:0.3-beta1, global installation with '''python
setup.py install''' is no longer required
* Open a terminal, enter the source folder ('''cd shotfactory''') and run
'''python shotfactory.py'''
See INSTALL for registration and troubleshooting tips.
\ No newline at end of file
pep8 :
pep8.py --filename=*.py --repeat .
pylint :
pylint shotfactory04 \
| grep -v "image/png\.py:" \
| grep -v "Method could be a function" \
| grep -v "defined outside __init__" \
| grep -v "Too many public methods"
docstrings :
-pylint shotserver04 \
| grep "docstring"
doctest :
grep -rl --include "*.py" "doctest.testmod()" . | xargs -n1 python
headers :
find . -name "*.py" \
| xargs header.py shotfactory04/__init__.py
properties :
find . -name "*.py" | xargs svn propset svn:keywords "Rev Date Author"
find . -type f | grep -v \\.svn | grep -v \\.log \
| grep -v \\.bz2 | grep -v \\.gz | grep -v \\.pyc \
| grep -v \\.png | grep -v \\.ico | grep -v \\.jpg | grep -v \\.ppm \
| xargs svn propset svn:eol-style native
documentation :
wget -O - http://trac.browsershots.org/wiki/InstallFactory?format=txt \
| sed s/InstallFactoryLinux/INSTALL.LINUX/ \
| sed s/InstallFactoryWindows/INSTALL.WIN/ \
| sed s/InstallFactoryMac/INSTALL.MAC/ \
| recode CR-LF..data | fold -s -w 76 > INSTALL
wget -O - http://trac.browsershots.org/wiki/InstallFactoryLinux?format=txt \
| sed s/InstallFactory/INSTALL/ \
| recode CR-LF..data | fold -s -w 76 > INSTALL.LINUX
wget -O - http://trac.browsershots.org/wiki/InstallFactoryWindows?format=txt \
| sed s/InstallFactory/INSTALL/ \
| recode CR-LF..data | fold -s -w 76 > INSTALL.WIN
wget -O - http://trac.browsershots.org/wiki/InstallFactoryMac?format=txt \
| sed s/InstallFactory/INSTALL/ \
| recode CR-LF..data | fold -s -w 76 > INSTALL.MAC
wget -O - http://trac.browsershots.org/wiki/ShotFactoryStandalone?format=txt \
| sed s/InstallFactory/INSTALL/ \
| recode CR-LF..data | fold -s -w 76 > STANDALONE
wget -O - http://trac.browsershots.org/wiki/FrequentlyAskedQuestions?format=txt \
| recode CR-LF..data | fold -s -w 76 > FAQ
wget -O - http://trac.browsershots.org/wiki/Authors?format=txt \
| recode CR-LF..data | fold -s -w 76 > AUTHORS
Browsershots
Copyright (C) 2007 Johann C. Rocholl <johann@browsershots.org>
Browsershots makes screenshots of your web design in different
browsers. It is a free open-source online service created by Johann C.
Rocholl. When you submit your web address, it will be added to the job
queue. A number of distributed computers will open your website in
their browser. Then they will make screenshots and upload them to the
central server.
This is the software for the distributed screenshot factories.
See INSTALL for installation instructions on Linux,
or INSTALL.WIN for Windows, or INSTALL.MAC for Mac OS X.
This software is licensed under the GNU GPL, version 3.
See COPYING for details.
= Running the !ShotFactory without a !ShotServer =
Since milestone:0.4-beta2 it is possible to run the ShotFactory without a
ShotServer installation.
== Create queue and output folder ==
Create two folders, one for the screenshot request queue, and one for the
resulting PNG files.
{{{
$ mkdir queue output
}}}
== Install the !ShotFactory and run it ==
See INSTALL for installation instructions.
When you run it, specify the queue and output folders on the command line.
The folders used below are examples. They can be anywhere, but the
ShotFactory must have read and write access to both of them.
{{{
shotfactory.py --queue /var/spool/shotfactory/queue --output /var/www/png
}}}
You can generate thumbnails too. The following example saves the original
PNG files, and also thumbnails that are 640 and 120 pixels wide, in the
output folders '''original''', '''large''' and '''small'''. These three
folders must already exist (in the current directory, because they're
specified as relative paths here). This feature requires the '''pnmscale'''
program from the '''netpbm''' package.
{{{
shotfactory.py --queue queue --output original --resize-output 640 large
--resize-output 120 small
}}}
If you only need the first screen of each website, you can limit the number
of pages to scroll down. This is useful for faster processing.
{{{
shotfactory.py --max-pages 1
}}}
To see a complete list of available options:
{{{
shotfactory.py --help
}}}
== Parallel operation ==
On Linux, you can run several screenshot factories on the same machine
(using separate user accounts).
If you store the queue and output folders on a NFS share, several machines
can work together. The file system must support atomic renames to avoid
race conditions. Otherwise, two factories may process the same screenshot
request.
== Request screenshots ==
Now you can request screenshots by creating files in the queue directory,
like this:
{{{
$ echo url http://www.example.com/ > queue/123
$ echo url http://www.example.org/ > queue/abc
}}}
If everything works, the ShotFactory will create the screenshot files
'''123.png''' and '''abc.png''' in the output folder.
Here's a more complete example for a screenshot request file:
{{{
request 123
url http://www.example.com/
width 1024
height 768
bpp 24
browser Firefox
command firefox
}}}
If the '''request''' parameter is given, its value will be the name of the
screenshot file, with '''.png''' appended. Otherwise, the name of the
screenshot request file will be used, also with '''.png''' added to it.
All other parameters except '''url''' are optional. Their default values
are as in the example above.
\ No newline at end of file
#!/usr/bin/env python
# browsershots.org - Test your web design in different browsers
# Copyright (C) 2007 Johann C. Rocholl <johann@browsershots.org>
#
# Browsershots is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# Browsershots is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Make screenshots and combine them into one tall image.
"""
__revision__ = "$Rev: 2006 $"
__date__ = "$Date: 2007-08-20 02:32:52 +0200 (Mon, 20 Aug 2007) $"
__author__ = "$Author: johann $"
import platform
import optparse
import time
def _main():
from optparse import OptionParser
version = '%prog ' + __revision__.strip('$').replace('Rev: ', 'r')
parser = OptionParser(version=version)
parser.add_option("-d", dest="display", action="store", type="string",
metavar="<name>", default=":0",
help="run on a different display (default :0)")
parser.add_option("-w", dest="wait", action="store", type="int",
metavar="<seconds>", default=5,
help="wait before taking screenshots (default 5)")
(options, args) = parser.parse_args()
config = {'width': 1024, 'bpp': 32}
system = platform.system()
if system == 'Linux':
from shotfactory04.gui import linux
gui = linux.Gui(config, options)
else:
raise NotImplemented(system)
print "Waiting %d seconds, please activate your browser window..." \
% options.wait
time.sleep(options.wait)
gui.browsershot()
if __name__ == '__main__':
_main()
#!/usr/bin/env python
# browsershots.org - Test your web design in different browsers
# Copyright (C) 2007 Johann C. Rocholl <johann@browsershots.org>
#
# Browsershots is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# Browsershots is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Find vertical offset between two PPM files.
"""
__revision__ = "$Rev: 2006 $"
__date__ = "$Date: 2007-08-20 02:32:52 +0200 (Mon, 20 Aug 2007) $"
__author__ = "$Author: johann $"
import sys
from shotfactory04.image import hashmatch
arg0, filename1, filename2 = sys.argv
print hashmatch.find_offset(filename1, filename2)
@echo off
del /S /Q build
rmdir build
del /S /Q bin
rmdir bin
python setup.py py2exe
copy thirdparty\*.* bin
# browsershots.org - Test your web design in different browsers
# Copyright (C) 2007 Johann C. Rocholl <johann@browsershots.org>
#
# Browsershots is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# Browsershots is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Installation script for use with distutils.
"""
__revision__ = "$Rev: 3047 $"
__date__ = "$Date: 2008-09-02 23:41:17 +0200 (Tue, 02 Sep 2008) $"
__author__ = "$Author: johann $"
from distutils.core import setup
import sys
kwargs = {
'name': 'ShotFactory',
'version': '0.4.0',
'description': 'Screenshot factory for browsershots.org',
'author': 'Johann C. Rocholl',
'author_email': 'johann@browsershots.org',
'url': 'http://v04.browsershots.org/',
'packages': [
'shotfactory04',
'shotfactory04.gui',
'shotfactory04.gui.darwin',
'shotfactory04.gui.linux',
'shotfactory04.gui.windows',
'shotfactory04.image',
'shotfactory04.servers',
],
'scripts': [
'shotfactory.py',
'browsershot.py',
'ppmoffset.py',
],
}
if 'py2exe' in sys.argv:
import py2exe
# modulefinder can't handle runtime changes to __path__,
# but win32com uses them
import modulefinder
import win32com
for path in win32com.__path__[1:]:
modulefinder.AddPackagePath("win32com", path)
__import__("win32com.shell")
m = sys.modules["win32com.shell"]
for path in m.__path__[1:]:
modulefinder.AddPackagePath("win32com.shell", path)
# py2exe configuration
kwargs['console'] = [{
'script': 'shotfactory.py',
'icon_resources': [(1, 'favicon.ico')],
}]
kwargs['options'] = {
'py2exe': {
'includes': ','.join([
'shotfactory04.gui.windows.firefox',
'shotfactory04.gui.windows.shiretoko',
'shotfactory04.gui.windows.chrome',
'shotfactory04.gui.windows.flock',
'shotfactory04.gui.windows.k_meleon',
'shotfactory04.gui.windows.msie',
'shotfactory04.gui.windows.minefield',
'shotfactory04.gui.windows.navigator',
'shotfactory04.gui.windows.opera',
'shotfactory04.gui.windows.safari',
'shotfactory04.gui.windows.seamonkey',
]),