[or-cvs] r15462: Added an interface for monitoring soat test results. Minor a (in torflow/branches/gsoc2008: . data/soat data/soat/http data/soat/ssh data/soat/ssl data/soat/ssl/certs)
aleksei at seul.org
aleksei at seul.org
Wed Jun 25 18:30:58 UTC 2008
Author: aleksei
Date: 2008-06-25 14:30:58 -0400 (Wed, 25 Jun 2008)
New Revision: 15462
Added:
torflow/branches/gsoc2008/data/soat/http/
torflow/branches/gsoc2008/data/soat/http/failed/
torflow/branches/gsoc2008/data/soat/http/inconclusive/
torflow/branches/gsoc2008/data/soat/http/successful/
torflow/branches/gsoc2008/data/soat/http/tags/
torflow/branches/gsoc2008/data/soat/ssh/
torflow/branches/gsoc2008/data/soat/ssh/failed/
torflow/branches/gsoc2008/data/soat/ssh/inconclusive/
torflow/branches/gsoc2008/data/soat/ssh/successful/
torflow/branches/gsoc2008/data/soat/ssl/failed/
torflow/branches/gsoc2008/data/soat/ssl/inconclusive/
torflow/branches/gsoc2008/data/soat/ssl/successful/
torflow/branches/gsoc2008/soatstats.py
Removed:
torflow/branches/gsoc2008/data/soat/ssl/certs/addons.mozilla.org.pem
torflow/branches/gsoc2008/data/soat/ssl/certs/mail.google.com.pem
torflow/branches/gsoc2008/data/soat/ssl/certs/www.fastmail.fm.pem
torflow/branches/gsoc2008/data/soat/ssl/certs/www.paypal.com.pem
torflow/branches/gsoc2008/data/soat/ssl/nodesPositive/
torflow/branches/gsoc2008/data/soat/ssl/nodesResults/
Modified:
torflow/branches/gsoc2008/soat.py
Log:
Added an interface for monitoring soat test results. Minor adjustments in the scanner.
Deleted: torflow/branches/gsoc2008/data/soat/ssl/certs/addons.mozilla.org.pem
===================================================================
--- torflow/branches/gsoc2008/data/soat/ssl/certs/addons.mozilla.org.pem 2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/data/soat/ssl/certs/addons.mozilla.org.pem 2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,19 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDAjCCAmugAwIBAgIDCG47MA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
-MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
-aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDcxMjEwMTgwMjMzWhcNMDkxMjEwMTgwMjMz
-WjCBjDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcT
-DU1vdW50YWluIFZpZXcxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xGjAY
-BgNVBAsTEVNlY3VyZSBXZWIgU2VydmVyMRYwFAYDVQQDFA0qLm1vemlsbGEub3Jn
-MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQChC0WH3YhyVVYKWEjAfYT0q19W
-c89J2McBEX9UvBUX5vOujghbDdyDTahVSf5sgkX54mVDoLq/rv28fd8Op8Wf54ZU
-oN/n+05M/F3+j52hja3UysHgtEsCCiTUd73oWTh0GiwNhbchqGWo52rs5xKGxaEQ
-HhW+gJFC2jrEYsRzbQIDAQABo4GuMIGrMA4GA1UdDwEB/wQEAwIE8DAdBgNVHQ4E
-FgQU441V52B+eQnnlTrlE0IcnvaS77QwOgYDVR0fBDMwMTAvoC2gK4YpaHR0cDov
-L2NybC5nZW90cnVzdC5jb20vY3Jscy9zZWN1cmVjYS5jcmwwHwYDVR0jBBgwFoAU
-SOZo+SvSspXXR9gjIBBPM5iQn9QwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
-BwMCMA0GCSqGSIb3DQEBBQUAA4GBAD7DR9FVBTK8jxLKebbAIetW95SIc9r1Nl5/
-PecjkP6RfKHbnCQ32j213HC9nZZR8nwCGgS1ryTKp/l0HFVWuF1L/y2fLEU7mzVP
-3cwNb/QYgOvNRh2+VDTP+rXGnLmm+MNvQoqzQNJrUlta8qxitaUFk/AL/anne8dp
-f+07Xq2p
------END CERTIFICATE-----
Deleted: torflow/branches/gsoc2008/data/soat/ssl/certs/mail.google.com.pem
===================================================================
--- torflow/branches/gsoc2008/data/soat/ssl/certs/mail.google.com.pem 2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/data/soat/ssl/certs/mail.google.com.pem 2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,19 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDIjCCAougAwIBAgIQbldpChBPqv+BdPg4iwgN8TANBgkqhkiG9w0BAQUFADBM
-MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
-THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wODA1MDIxNjMyNTRaFw0w
-OTA1MDIxNjMyNTRaMGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
-MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRgw
-FgYDVQQDEw9tYWlsLmdvb2dsZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJ
-AoGBALlkxdh2QXegdElukCSOV2+8PKiONIS+8Tu9K7MQsYpqtLNC860zwOPQ2NLI
-3Zp4jwuXVTrtzGuiqf5Jioh35Ig3CqDXtLyZoypjZUQcq4mlLzHlhIQ4EhSjDmA7
-Ffw9y3ckSOQgdBQWNLbquHh9AbEUjmhkrYxIqKXeCnRKhv6nAgMBAAGjgecwgeQw
-KAYDVR0lBCEwHwYIKwYBBQUHAwEGCCsGAQUFBwMCBglghkgBhvhCBAEwNgYDVR0f
-BC8wLTAroCmgJ4YlaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVNHQ0NBLmNy
-bDByBggrBgEFBQcBAQRmMGQwIgYIKwYBBQUHMAGGFmh0dHA6Ly9vY3NwLnRoYXd0
-ZS5jb20wPgYIKwYBBQUHMAKGMmh0dHA6Ly93d3cudGhhd3RlLmNvbS9yZXBvc2l0
-b3J5L1RoYXd0ZV9TR0NfQ0EuY3J0MAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEF
-BQADgYEAsRwpLg1dgCR1gYDK185MFGukXMeQFUvhGqF8eT/CjpdvezyKVuz84gSu
-6ccMXgcPQZGQN/F4Xug+Q01eccJjRSVfdvR5qwpqCj+6BFl5oiKDBsveSkrmL5dz
-s2bn7TdTSYKcLeBkjXxDLHGBqLJ6TNCJ3c4/cbbG5JhGvoema94=
------END CERTIFICATE-----
Deleted: torflow/branches/gsoc2008/data/soat/ssl/certs/www.fastmail.fm.pem
===================================================================
--- torflow/branches/gsoc2008/data/soat/ssl/certs/www.fastmail.fm.pem 2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/data/soat/ssl/certs/www.fastmail.fm.pem 2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,21 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDfDCCAuWgAwIBAgIQbfxRUFCOzMrPD9Y7lr8MaTANBgkqhkiG9w0BAQUFADCB
-zjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJ
-Q2FwZSBUb3duMR0wGwYDVQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UE
-CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEhMB8GA1UEAxMYVGhh
-d3RlIFByZW1pdW0gU2VydmVyIENBMSgwJgYJKoZIhvcNAQkBFhlwcmVtaXVtLXNl
-cnZlckB0aGF3dGUuY29tMB4XDTA4MDUxNDIyMjkyMloXDTA5MDUyMDA5MjE0Nlow
-gYAxCzAJBgNVBAYTAkFVMRgwFgYDVQQIEw9OZXcgU291dGggV2FsZXMxEzARBgNV
-BAcTCkNyb3dzIE5lc3QxKDAmBgNVBAoTH09wdGltYWwgRGVjaXNpb25zIEdyb3Vw
-IFB0eSBMdGQxGDAWBgNVBAMTD3d3dy5mYXN0bWFpbC5mbTCBnzANBgkqhkiG9w0B
-AQEFAAOBjQAwgYkCgYEAyJMTIn/14CHIrAYGi3913dMk1khF1C1M4f6/JlmIv7Xl
-HlOT+GJNcal38wsS/VxXTSrKgPcXDDK0kYGg/1WU6wF8HJ64LEXHou558PDBDnnp
-4pw2ayB5mVy9E3YFez0a/NHEBDKrkngmQmWQxmcVzBLNDYhoR/NQPQ6JZjtR1kMC
-AwEAAaOBpjCBozAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwQAYDVR0f
-BDkwNzA1oDOgMYYvaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVByZW1pdW1T
-ZXJ2ZXJDQS5jcmwwMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzABhhZodHRwOi8v
-b2NzcC50aGF3dGUuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEA
-IKus8RSbHoHfE36J44uEyGPMxFnfaEZLoZs+k6aZPnfVyvhDwNKPmfoEkaFWtqQ1
-Q+5IzVWGXCQOd5R4bWiiJRhG33KCDmRfqnVrCX0X+V6N72e9Zw51ac1Tv+Noipan
-s/yFWHIOod8fw6ELGcXzOd5mEt5XlCFWm/2IKtRCG9E=
------END CERTIFICATE-----
Deleted: torflow/branches/gsoc2008/data/soat/ssl/certs/www.paypal.com.pem
===================================================================
--- torflow/branches/gsoc2008/data/soat/ssl/certs/www.paypal.com.pem 2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/data/soat/ssl/certs/www.paypal.com.pem 2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,34 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIF5jCCBM6gAwIBAgIQbmuco/dSNbSVN4bU5RNUqTANBgkqhkiG9w0BAQUFADCB
-vjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
-ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug
-YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykwNjE4MDYGA1UEAxMv
-VmVyaVNpZ24gQ2xhc3MgMyBFeHRlbmRlZCBWYWxpZGF0aW9uIFNTTCBTR0MgQ0Ew
-HhcNMDcwMTMwMDAwMDAwWhcNMDkwMTI5MjM1OTU5WjCCAR4xEDAOBgNVBAUTBzMw
-MTQyNjcxEzARBgsrBgEEAYI3PAIBAxMCVVMxGTAXBgsrBgEEAYI3PAIBAhMIRGVs
-YXdhcmUxCzAJBgNVBAYTAlVTMRMwEQYDVQQRFAo5NTEzMS0yMDIxMQswCQYDVQQI
-EwJDQTERMA8GA1UEBxQIU2FuIEpvc2UxFjAUBgNVBAkUDTIyMTEgTiAxc3QgU3Qx
-FDASBgNVBAoUC1BheXBhbCBJbmMuMRwwGgYDVQQLFBNJbmZvcm1hdGlvbiBTeXN0
-ZW1zMTMwMQYDVQQLFCpUZXJtcyBvZiB1c2UgYXQgd3d3LnZlcmlzaWduLmNvbS9y
-cGEgKGMpMDYxFzAVBgNVBAMUDnd3dy5wYXlwYWwuY29tMIGfMA0GCSqGSIb3DQEB
-AQUAA4GNADCBiQKBgQC0ldZuxq9UVX26JfEnMM+U7pM+husarblH+rS18AyT8Aqf
-0oOZJ+439TESAQ521/9zDqXYpm5++VoNN5/M3HbQ63ksuqDILhA+G1sU0jA9RD38
-z992oazPJQDTfmeK+amv50+4okUT8QTDkb9WHjUI/gpf14AYQ628UFQQGO6WlQID
-AQABo4IB/zCCAfswCQYDVR0TBAIwADAdBgNVHQ4EFgQU7VBMXqBNGpJwm/AUUdLW
-nWziydAwCwYDVR0PBAQDAgWgMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9FVklu
-dGwtY3JsLnZlcmlzaWduLmNvbS9FVkludGwyMDA2LmNybDBEBgNVHSAEPTA7MDkG
-C2CGSAGG+EUBBxcGMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LnZlcmlzaWdu
-LmNvbS9ycGEwNAYDVR0lBC0wKwYIKwYBBQUHAwEGCCsGAQUFBwMCBglghkgBhvhC
-BAEGCisGAQQBgjcKAwMwHwYDVR0jBBgwFoAUTkPIHXbvN1N6T/JYb5TzOOLVvd8w
-dgYIKwYBBQUHAQEEajBoMCsGCCsGAQUFBzABhh9odHRwOi8vRVZJbnRsLW9jc3Au
-dmVyaXNpZ24uY29tMDkGCCsGAQUFBzAChi1odHRwOi8vRVZJbnRsLWFpYS52ZXJp
-c2lnbi5jb20vRVZJbnRsMjAwNi5jZXIwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcw
-VRYJaW1hZ2UvZ2lmMCEwHzAHBgUrDgMCGgQUj+XTGoasjY5rw8+AatRIGCx7GS4w
-JRYjaHR0cDovL2xvZ28udmVyaXNpZ24uY29tL3ZzbG9nby5naWYwDQYJKoZIhvcN
-AQEFBQADggEBADGj5/xJrl7mzkgcE6zf44jzT0Iw+/BLAzcerNixC3xZ57rPPhxu
-WRMKVzgIDYQzu/ed2oLRz6eefjF2Xs9XqLK1RDYRyDldksxP/AxB52AEGoBbufQ1
-xKWziRJXaUmqA27FBbx89lYrj3m6VQkkJ7bK+IpP9vQzpg86ZBQs0UWtxsQbO9zp
-D8IaeI+K97d5CgRfXixgfUR3OttmhofzmR9Zfawcui+MoEu0lrjeZNgAkEaCVlcH
-m0iL1rOpmhX/7b5DUpTWN3jopw9/VQTCqxcu+0zoEcNU02yadEVCNumWAEllTJ5D
-ePLh8FkejTsIkNexlknGZX4eQJYBo90wUok=
------END CERTIFICATE-----
Modified: torflow/branches/gsoc2008/soat.py
===================================================================
--- torflow/branches/gsoc2008/soat.py 2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/soat.py 2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,5 +1,6 @@
#!/usr/bin/python
import httplib
+import md5
import os
import random
import re
@@ -12,8 +13,10 @@
import urllib
import urllib2
+import soatstats
+from soatstats import *
+
from TorCtl import TorUtil, TorCtl, PathSupport
-
from TorCtl.TorUtil import meta_port, meta_host, control_port, control_host, tor_port, tor_host
from TorCtl.TorUtil import *
from TorCtl.PathSupport import *
@@ -28,7 +31,7 @@
# Try to use system openssl first
try:
from OpenSSL import *
-except e:
+except:
sys.path.append("./tools/")
from OpenSSL import *
@@ -47,15 +50,16 @@
same_origin_policy = True
-ssl_certs_directory = './data/soat/ssl/certs/';
-ssl_nodes_results_directory = './data/soat/ssl/nodesResults/';
-ssl_nodes_positive_directory = './data/soat/ssl/nodesPositive/';
+#
+# data storage
+#
+
#
# links of interest
#
-docs_http = ['http://www.torproject.org']
+docs_http = ['http://www.torproject.org','http://www.math.ut.ee','http://www.mozilla.com']
docs_https = ['mail.google.com','addons.mozilla.org','www.paypal.com','www.fastmail.fm']
#
@@ -98,21 +102,6 @@
s = s[:-1]
return s
-# a class for saving ssl test results
-class OpenSSLTestResult:
- def __init__(self, exit_node, ssl_site, cert_file):
- self.exit_node = exit_node
- self.ssl_site = ssl_site
- self.timestamp = time.time()
- self.cert = cert_file
-
-# a class for saving http test results
-class HttpTestResult:
- def __init__(self, exit_node, website):
- self.exit_node = exit_node
- self.website = website
- self.timestamp = time.time()
-
# The scanner class
class ExitNodeScanner:
@@ -222,7 +211,8 @@
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, tor_host, tor_port)
socket.socket = socks.socksocket
- plog('INFO', 'Opening ' + address + ' using exit node ' + self.get_exit_node())
+ exit_node = self.get_exit_node()
+ plog('INFO', 'Opening ' + address + ' using exit node ' + exit_node)
try:
g = urllib2.urlopen(request)
except Exception, e:
@@ -232,7 +222,7 @@
return 0
pcontent = g.read()
- pcontent.decode('ascii', 'ignore')
+ pcontent = pcontent.decode('ascii', 'ignore')
node_page = BeautifulSoup(pcontent)
@@ -241,7 +231,27 @@
# nasty tags: a, applet, div, embed, form, frame, iframe, img, link, script
# also check DOM event stuff
+
+ # address[7:] means we're leaving out the 'http://' from the address part
+ #
+ # just some temporary stuff (compare the whole content) that will be replaced soon
+ #
+ if md5.new(pcontent).digest() != md5.new(content).digest():
+ plog('ERROR', 'The retrieved pages dont match')
+
+ content_file_handle = open(http_tags_dir + address[7:] + '.result','w')
+ content_file_handle.write(content)
+ content_file_handle.close()
+ pcontent_file_handle = open(http_tags_dir + `exit_node` + '_' + address[7:] + '.result','w')
+ pcontent_file_handle.write(pcontent)
+ pcontent_file_handle.close()
+
+ else:
+ result = HttpTestResult(exit_node, address, 0, TEST_SUCCESS)
+ result_file = open(http_s_dir + `exit_node` + '_' + address[7:] + '.result','w')
+ pickle.dump(result, result_file)
+
return 0
def check_openssh(self, address):
@@ -268,16 +278,21 @@
c.set_connect_state()
plog('INFO', 'Opening a direct ssl connection to ' + address)
+
+ try:
+ c.connect((address, 443))
+ c.send(crypto.dump_certificate_request(crypto.FILETYPE_ASN1,request))
+ except socket.error, e:
+ plog('ERROR','An error occured while opening a direct ssl connection to ' + address)
+ plog('ERROR', e)
+ return 0
- c.connect((address, 443))
- c.send(crypto.dump_certificate_request(crypto.FILETYPE_ASN1,request))
-
cert = c.get_peer_certificate()
cert_pem = crypto.dump_certificate(crypto.FILETYPE_PEM, cert)
# save the cert
- cert_file_handle = open(ssl_certs_directory + address + '.pem', 'w')
+ cert_file_handle = open(ssl_certs_dir + address + '.pem', 'w')
cert_file_handle.write(cert_pem)
cert_file_handle.close()
@@ -289,7 +304,7 @@
# check whether we already have a circuit.
# if yes, open a connection via tor, otherwise skip to the next test
exit_node = self.get_exit_node()
- if exit_node == 0:
+ if exit_node == 0 or exit_node == '0':
plog('INFO', 'We have no exit node to test, skipping to the next test.')
return 0
@@ -303,8 +318,23 @@
plog('INFO', 'Opening an ssl connection to ' + address + ' using exit node ' + `exit_node`)
- c2.connect((address, 443))
- c2.send(crypto.dump_certificate_request(crypto.FILETYPE_ASN1,request))
+ try:
+ c2.connect((address, 443))
+ except Exception, e:
+ plog('ERROR','An error occured while opening a direct ssl connection to ' + address)
+ plog('ERROR', e)
+ return 0
+
+ try:
+ c2.send(crypto.dump_certificate_request(crypto.FILETYPE_ASN1,request))
+ except Exception, e:
+ plog('ERROR', 'Got the following ssl error while connecting via tor:')
+ plog('ERROR', e)
+ cert_file = ssl_certs_dir + address + '.pem'
+ result = OpenSSLTestResult(exit_node, address, cert_file, TEST_INCONCLUSIVE)
+ result_file = open(ssl_i_dir + `exit_node` + '_' + address + '.result','w')
+ pickle.dump(result, result_file)
+ return 0
cert2 = c2.get_peer_certificate()
cert2_pem = crypto.dump_certificate(crypto.FILETYPE_PEM, cert2)
@@ -317,20 +347,20 @@
# save the received cert for inspection
if cert_pem == cert2_pem:
- cert_file = ssl_certs_directory + address + '.pem'
- result = OpenSSLTestResult(exit_node, address, cert_file)
- result_file = open(ssl_nodes_results_directory + `exit_node` + '_' + address + '.result','w')
+ cert_file = ssl_certs_dir + address + '.pem'
+ result = OpenSSLTestResult(exit_node, address, cert_file, TEST_SUCCESS)
+ result_file = open(ssl_s_dir + `exit_node` + '_' + address + '.result','w')
pickle.dump(result, result_file)
else:
plog('ERROR', 'Exit node ' + `exit_node` + ' seems to be meddling with certificates. (' + address + ')')
- cert_file = ssl_certs_directory + address + '_' + `exit_node` + '.pem'
+ cert_file = ssl_certs_dir + address + '_' + `exit_node` + '.pem'
cert_file_handle = open(cert_file, 'w')
cert_file_handle.write(cert2_pem)
cert_file_handle.close()
- result = OpenSSLTestResult(exit_node, address, cert_file)
- result_file = open(ssl_nodes_positive_directory + `exit_node` + '_' + address + '.result','w')
+ result = OpenSSLTestResult(exit_node, address, cert_file, TEST_FAILURE)
+ result_file = open(ssl_f_dir + `exit_node` + '_' + address + '.result','w')
pickle.dump(result, result_file)
plog('INFO', 'Test complete. Moving on...')
@@ -370,18 +400,6 @@
return urllist
-def load_cert():
- filehandler = open('./data/soat/ssl/certs/addons.mozilla.org.pem','r')
- string = filehandler.read()
-
- ctx = SSL.Context(SSL.SSLv23_METHOD)
- ctx.use_certificate_file('./data/soat/ssl/certs/addons.mozilla.org.pem')
-
- cert = crypto.load_certificate(crypto.FILETYPE_PEM, string)
- print cert.get_subject()
-
- return 0
-
'''
Find links to files related to a query
'''
@@ -433,6 +451,8 @@
# consistency test
# scanner.check_all_exits_port_consistency()
+ while 1:
+
# find sites for http testing if necessary
#
# global doc_urls
@@ -441,12 +461,12 @@
# plog('NOTICE', 'Final URL list: ' + '\n'.join(doc_urls) + '\n')
# https test
- for ssl_site in docs_https:
- scanner.check_openssl(ssl_site)
-
+ for ssl_site in docs_https:
+ scanner.check_openssl(ssl_site)
+
# http test
- # for http_site in docs_http:
- # scanner.check_http(http_site)
+ for http_site in docs_http:
+ scanner.check_http(http_site)
#
# initiate the program
Added: torflow/branches/gsoc2008/soatstats.py
===================================================================
--- torflow/branches/gsoc2008/soatstats.py (rev 0)
+++ torflow/branches/gsoc2008/soatstats.py 2008-06-25 18:30:58 UTC (rev 15462)
@@ -0,0 +1,440 @@
+import operator
+import os
+import pickle
+import sys
+import time
+
+import sets
+from sets import Set
+
+#
+# Data storage
+#
+
+# data locations
+
+data_dir = './data/soat/'
+
+ssl_certs_dir = data_dir + 'ssl/certs/'
+ssl_s_dir = data_dir + 'ssl/successful/'
+ssl_f_dir = data_dir + 'ssl/failed/'
+ssl_i_dir = data_dir + 'ssl/inconclusive/'
+
+http_tags_dir = data_dir + 'http/tags/'
+http_s_dir = data_dir + 'http/successful/'
+http_f_dir = data_dir + 'http/failed/'
+http_i_dir = data_dir + '/http/inconclusive/'
+
+ssh_s_dir = data_dir + 'ssh/successful/'
+ssh_f_dir = data_dir + 'ssh/failed/'
+ssh_i_dir = data_dir + 'ssh/inconclusive/'
+
+result_dirs = [ssl_s_dir, ssl_f_dir, ssl_i_dir,
+ http_s_dir, http_f_dir, http_i_dir]
+
+# classes to use with pickle to dump test results into files
+
+TEST_SUCCESS = 0
+TEST_INCONCLUSIVE = 1
+TEST_FAILURE = 2
+
+class TestResult(object):
+ def __init__(self, exit_node, site, status):
+ self.exit_node = exit_node
+ self.site = site
+ self.timestamp = time.time()
+ self.status = status
+
+class OpenSSLTestResult(TestResult):
+ def __init__(self, exit_node, ssl_site, cert_file, status):
+ super(OpenSSLTestResult, self).__init__(exit_node, ssl_site, status)
+ self.cert = cert_file
+
+class HttpTestResult(TestResult):
+ def __init__(self, exit_node, website, tag_prints, status):
+ super(HttpTestResult, self).__init__(exit_node, website, status)
+ self.tag_prints = tag_prints
+
+class OpenSSHTestResult(TestResult):
+ def __init__(self, exit_node, ssh_site, status):
+ super(OpenSSHTestResult, self).__init__(exit_node, ssh_site, status)
+
+#
+# Class for saving and managing data
+#
+class DataHandler():
+
+ # function for filtering results based on protocol and success level
+ def filterResults(self, results,
+ show_ssh, show_http, show_ssl,
+ show_good, show_bad, show_unsure):
+
+ filters = []
+ if not show_ssh:
+ filters.append(lambda x: not x.__class__.__name__ == 'OpenSSHTestResult')
+ if not show_ssl:
+ filters.append(lambda x: not x.__class__.__name__ == 'OpenSSLTestResult')
+ if not show_http:
+ filters.append(lambda x: not x.__class__.__name__ == 'HttpTestResult')
+ if not show_good:
+ filters.append(lambda x: not x.status == TEST_SUCCESS)
+ if not show_bad:
+ filters.append(lambda x: not x.status == TEST_FAILURE)
+ if not show_unsure:
+ filters.append(lambda x: not x.status == TEST_INCONCLUSIVE)
+
+ filtered = []
+ if len(filters) > 0:
+ filter = lambda x: reduce(operator.__and__, [f(x) for f in filters])
+ filtered = [x for x in results if filter(x)]
+ else:
+ filtered = list(results)
+
+ return filtered
+
+ # get all available results
+ def getAll(self):
+ results = []
+
+ for dir in result_dirs:
+ files = os.listdir(dir)
+ for file in files:
+ fh = open(dir + file,'r')
+ result = pickle.load(fh)
+ results.append(result)
+
+ return results
+
+ # filter by node
+ def filterByNode(self, results, id):
+ return filter(lambda x: x.exit_node == id, results)
+
+#
+# Displaying stats on the console
+#
+
+#
+# Class to display statistics on the console
+#
+# It can show the summary
+#
+class StatsConsole():
+ def Listen(self):
+ while 1:
+ input = raw_input(">>>")
+ if input == 'e' or input == 'exit':
+ exit()
+ elif input == 's' or input == 'summary':
+ self.Summary()
+ elif input == 'h' or input == 'help' or len(input) > 6:
+ self.Help()
+ else:
+ self.Reply(input)
+
+ def Summary(self):
+ dh = DataHandler()
+ data = dh.getAll()
+ nodeSet = Set([])
+
+ total = len(data)
+ good = 0
+ bad = 0
+ inconclusive = 0
+ ssh = 0
+ http = 0
+ ssl = 0
+
+ for result in data:
+ nodeSet.add(result.exit_node)
+
+ if result.status == 0:
+ good += 1
+ elif result.status == 1:
+ inconclusive += 1
+ elif result.status == 2:
+ bad += 1
+
+ if result.__class__.__name__ == 'OpenSSHTestResult':
+ ssh += 1
+ elif result.__class__.__name__ == 'HttpTestResult':
+ http += 1
+ elif result.__class__.__name__ == 'OpenSSLTestResult':
+ ssl += 1
+
+ swidth = 25
+ nwidth = 10
+ width = swidth + nwidth
+
+ header_format = '%-*s%*s'
+ format = '%-*s%*i'
+
+ print '=' * width
+ print header_format % (swidth, 'Parameter', nwidth, 'Count')
+ print '-' * width
+
+ print format % (swidth, 'Tests completed', nwidth, total)
+ print format % (swidth, 'Nodes tested', nwidth, len(nodeSet))
+ print format % (swidth, 'Failed tests', nwidth, bad)
+ print format % (swidth, 'Succeeded tests', nwidth, good)
+ print format % (swidth, 'Inconclusive tests', nwidth, inconclusive)
+ print format % (swidth, 'SSH tests', nwidth, ssh)
+ print format % (swidth, 'HTTP tests', nwidth, http)
+ print format % (swidth, 'SSL tests', nwidth, ssl)
+
+ print '=' * width
+
+ def Reply(self, input):
+
+ ssh = False
+ http = False
+ ssl = False
+ good = False
+ bad = False
+ inconclusive = False
+
+ if 'a' in input:
+ ssh = http = ssl = good = bad = inconclusive = True
+ else:
+ if 's' in input:
+ ssh = True
+ if 'h' in input:
+ http = True
+ if 'l' in input:
+ ssl = True
+ if 'g' in input:
+ good = True
+ if 'b' in input:
+ bad = True
+ if 'i' in input:
+ inconclusive = True
+
+ dh = DataHandler()
+ data = dh.getAll()
+ filtered = dh.filterResults(data, ssh, http, ssl, good, bad, inconclusive)
+
+ nodewidth = 45
+ typewidth = 10
+ sitewidth = 30
+ timewidth = 30
+ statuswidth = 6
+ width = nodewidth + typewidth + sitewidth + timewidth + statuswidth
+
+ format = '%-*s%-*s%-*s%-*s%-*s'
+
+ print '=' * width
+ print format % (nodewidth, 'Exit node', typewidth, 'Test type', sitewidth, 'Remote site',
+ timewidth, 'Time', statuswidth, 'Status')
+ print '-' * width
+ for result in filtered:
+ print format % (nodewidth, `result.exit_node`,
+ typewidth, result.__class__.__name__[:-10],
+ sitewidth, result.site,
+ timewidth, time.strftime("%a, %d %b %Y %H:%M:%S", time.localtime(result.timestamp)),
+ statuswidth, `result.status`)
+ print '=' * width
+
+ def Help(self):
+ print ''
+ print 'Options:'
+ print '* summmary (s) - display a short summary about all tests done so far'
+ print '* exit (e) - terminate the program'
+ print '* help (h) - display this help text'
+ print '* all (a) - list all the results'
+ print '* (shlgbi) - display a filtered list of test results. Letters are optional and mean the following:'
+ print ' s - show ssh results'
+ print ' h - show http results'
+ print ' l - show ssl results'
+ print ' g - show good results'
+ print ' b - show bad results'
+ print ' i - show inconclusive results'
+ print ''
+
+#
+# Displaying stats in a graphical setting (first check if we have wx)
+#
+
+nowx = False
+try:
+ import wx
+ from wx.lib.mixins.listctrl import ListCtrlAutoWidthMixin, ColumnSorterMixin
+except:
+ nowx = True
+
+if not nowx:
+
+ # The list model to be used for the result listing
+ class ListMixin(wx.ListCtrl, ListCtrlAutoWidthMixin, ColumnSorterMixin):
+ def __init__(self, parent, map):
+ wx.ListCtrl.__init__(self, parent, -1, style=wx.LC_REPORT)
+ ListCtrlAutoWidthMixin.__init__(self)
+ ColumnSorterMixin.__init__(self, len(map))
+ self.itemDataMap = map
+
+ def GetListCtrl(self):
+ return self
+
+ # menu item ids
+ ID_EXIT = 1
+
+ ID_SHOW_GOOD = 11
+ ID_SHOW_BAD = 12
+ ID_SHOW_UNSURE = 13
+
+ ID_SHOW_SSL = 21
+ ID_SHOW_HTTP = 22
+ ID_SHOW_SSH = 23
+
+ ID_NODE = 31
+
+ # the main application window
+ class MainFrame(wx.Frame):
+ def __init__(self):
+ wx.Frame.__init__(self, None, title="Soat test results", size=(900,500))
+
+ # get the data
+
+ self.dataHandler = DataHandler()
+ self.dataList = self.dataHandler.getAll()
+ self.filteredList = self.dataList
+
+ # display it
+
+ self.CreateStatusBar()
+ self.initMenuBar()
+ self.initContent()
+
+ self.Center()
+ self.Show()
+
+ def initMenuBar(self):
+ fileMenu = wx.Menu()
+ fileMenu.Append(ID_EXIT, "E&xit", "Exit the program")
+
+ viewMenu = wx.Menu()
+ self.showGood = viewMenu.Append(ID_SHOW_GOOD, 'Show &Good', 'Show sucessful test results', kind=wx.ITEM_CHECK)
+ self.showBad = viewMenu.Append(ID_SHOW_BAD, 'Show &Bad', 'Show unsucessful test results', kind=wx.ITEM_CHECK)
+ self.showUnsure = viewMenu.Append(ID_SHOW_UNSURE, 'Show &Inconclusive', 'Show inconclusive test results', kind=wx.ITEM_CHECK)
+ viewMenu.AppendSeparator()
+ self.showSSL = viewMenu.Append(ID_SHOW_SSL, 'Show SS&L', 'Show SSL test results', kind=wx.ITEM_CHECK)
+ self.showHTTP = viewMenu.Append(ID_SHOW_HTTP, 'Show &HTTP', 'Show HTTP test results', kind=wx.ITEM_CHECK)
+ self.showSSH = viewMenu.Append(ID_SHOW_SSH, 'Show &SSH', 'Show SSH test results', kind=wx.ITEM_CHECK)
+ viewMenu.AppendSeparator()
+ viewMenu.Append(ID_NODE, '&Find node...', 'View test results for a given node [NOT IMPLEMENTED]')
+
+ viewMenu.Check(ID_SHOW_GOOD, True)
+ viewMenu.Check(ID_SHOW_BAD, True)
+ viewMenu.Check(ID_SHOW_UNSURE, True)
+ viewMenu.Check(ID_SHOW_SSL, True)
+ viewMenu.Check(ID_SHOW_HTTP, True)
+ viewMenu.Check(ID_SHOW_SSH, True)
+
+ menuBar = wx.MenuBar()
+ menuBar.Append(fileMenu,"&File")
+ menuBar.Append(viewMenu,"&View")
+
+ self.SetMenuBar(menuBar)
+
+ wx.EVT_MENU(self, ID_EXIT, self.OnExit)
+
+ wx.EVT_MENU(self, ID_SHOW_GOOD, self.GenerateFilteredList)
+ wx.EVT_MENU(self, ID_SHOW_BAD, self.GenerateFilteredList)
+ wx.EVT_MENU(self, ID_SHOW_UNSURE, self.GenerateFilteredList)
+ wx.EVT_MENU(self, ID_SHOW_SSL, self.GenerateFilteredList)
+ wx.EVT_MENU(self, ID_SHOW_HTTP, self.GenerateFilteredList)
+ wx.EVT_MENU(self, ID_SHOW_SSH, self.GenerateFilteredList)
+
+ def initContent(self):
+
+ base = wx.Panel(self, -1)
+ sizer = wx.GridBagSizer(0,0)
+
+ box = wx.StaticBox(base, -1, 'Summary')
+ boxSizer = wx.StaticBoxSizer(box, wx.HORIZONTAL)
+
+ total = wx.StaticText(base, -1, 'Total tests: ' + `len(self.filteredList)`)
+ boxSizer.Add(total, 0, wx.LEFT | wx.TOP | wx.BOTTOM, 10)
+
+ nodes = wx.StaticText(base, -1, 'Nodes scanned: ' + `len(Set([x.exit_node for x in self.filteredList]))`)
+ boxSizer.Add(nodes, 0, wx.LEFT | wx.TOP | wx.BOTTOM , 10)
+
+ bad = wx.StaticText(base, -1, 'Failed tests: ' + `len([x for x in self.filteredList if x.status == 2])`)
+ boxSizer.Add(bad, 0, wx.LEFT | wx.TOP | wx.BOTTOM, 10)
+
+ suspicious = wx.StaticText(base, -1, 'Inconclusive tests: ' + `len([x for x in self.filteredList if x.status == 1])`)
+ boxSizer.Add(suspicious, 0, wx.ALL, 10)
+
+ sizer.Add(boxSizer, (0,0), (1, 5), wx.EXPAND | wx.ALL, 15)
+
+ dataMap = {}
+ self.fillDataMap(dataMap)
+
+ self.listCtrl = ListMixin(base, dataMap)
+ self.listCtrl.InsertColumn(0, 'exit node', width=380)
+ self.listCtrl.InsertColumn(1, 'type', width=70)
+ self.listCtrl.InsertColumn(2, 'site', width=180)
+ self.listCtrl.InsertColumn(3, 'time', width=180)
+ self.listCtrl.InsertColumn(4, 'status', wx.LIST_FORMAT_CENTER, width=50)
+
+ self.fillListCtrl(dataMap)
+
+ sizer.Add(self.listCtrl, (1,0), (1,5), wx.EXPAND | wx.LEFT | wx.BOTTOM | wx.RIGHT, border=15)
+
+ sizer.AddGrowableCol(3)
+ sizer.AddGrowableRow(1)
+
+ base.SetSizerAndFit(sizer)
+
+ # make a nasty dictionary from the current self.filteredList object so columns would be sortable
+ def fillDataMap(self, dataMap):
+ for i in range(len(self.filteredList)):
+ dataMap.update([(i,(self.filteredList[i].exit_node,
+ self.filteredList[i].__class__.__name__[:-10],
+ self.filteredList[i].site,
+ time.strftime("%a, %d %b %Y %H:%M:%S", time.localtime(self.filteredList[i].timestamp)),
+ self.filteredList[i].status))])
+
+ # fill the result listing with data
+ def fillListCtrl(self, dataMap):
+ if self.listCtrl.GetItemCount() > 0:
+ self.listCtrl.DeleteAllItems()
+
+ for k, i in dataMap.items():
+ index = self.listCtrl.InsertStringItem(sys.maxint, `i[0]`)
+ self.listCtrl.SetStringItem(index, 1, i[1])
+ self.listCtrl.SetStringItem(index, 2, `i[2]`)
+ self.listCtrl.SetStringItem(index, 3, i[3])
+ self.listCtrl.SetStringItem(index, 4, `i[4]`)
+ self.listCtrl.SetItemData(index,k)
+
+ def OnExit(self,e):
+ self.Close(True)
+
+ def GenerateFilteredList(self, e):
+ self.filteredList = list(self.dataHandler.filterResults(self.dataList,
+ self.showSSH.IsChecked(), self.showHTTP.IsChecked(), self.showSSL.IsChecked(),
+ self.showGood.IsChecked(), self.showBad.IsChecked(), self.showUnsure.IsChecked()))
+ dataMap = {}
+ self.fillDataMap(dataMap)
+ self.fillListCtrl(dataMap)
+ self.listCtrl.RefreshItems(0, len(dataMap))
+
+if __name__ == "__main__":
+ if len(sys.argv) == 1:
+ console = StatsConsole()
+ console.Listen()
+ elif len(sys.argv) == 2 and sys.argv[1] == 'wx':
+ if nowx:
+ print 'wxpython doesn\'t seem to be installed on your system'
+ print 'you can use the console interface instead (see help)'
+ else:
+ app = wx.App(0)
+ MainFrame()
+ app.MainLoop()
+ else:
+ print ''
+ print 'This app displays results of tests carried out by soat.py (in a user-friendly way).'
+ print ''
+ print 'Usage:'
+ print 'python soatstats.py - app starts console-only'
+ print 'python soatstats.py wx - app starts with a wxpython gui'
+ print ''
Property changes on: torflow/branches/gsoc2008/soatstats.py
___________________________________________________________________
Name: svn:executable
+ *
More information about the tor-commits
mailing list