[tor-bugs] #14177 [Torflow]: bwauth spends some 40 minutes "updating rank history". Every hour.
Tor Bug Tracker & Wiki
blackhole at torproject.org
Wed Feb 4 22:03:04 UTC 2015
#14177: bwauth spends some 40 minutes "updating rank history". Every hour.
-------------------------+--------------------
Reporter: ln5 | Owner: aagbsn
Type: defect | Status: new
Priority: major | Milestone:
Component: Torflow | Version:
Resolution: | Keywords:
Actual Points: | Parent ID:
Points: |
-------------------------+--------------------
Comment (by cypherpunks):
In the hopes that this will help debugging, here is the configuration from
a debian wheezy system
{{{
dpkg -l | egrep python
ii libpython2.7 2.7.3-6+deb7u2 amd64
Shared Python runtime library (version 2.7)
ii python 2.7.3-4+deb7u1 all
interactive high-level object-oriented language (default version)
ii python-apt 0.8.8.2 amd64
Python interface to libapt-pkg
ii python-apt-common 0.8.8.2 all
Python interface to libapt-pkg (locales)
ii python-central 0.6.17 all
register and build utility for Python packages
ii python-crypto 2.6-4+deb7u3 amd64
cryptographic algorithms and protocols for Python
ii python-dev 2.7.3-4+deb7u1 all
header files and a static library for Python (default)
ii python-dumbnet 1.12-3.1 amd64
A dumb, portable networking library -- python bindings
ii python-elixir 0.7.1-1 all
Declarative Mapper for SQLAlchemy
ii python-geoip 1.3.2-1~bpo70+1 amd64
Python bindings for the GeoIP IP-to-country resolver library
ii python-httplib2 0.7.4-2+deb7u1 all
comprehensive HTTP client library written for Python
ii python-ipaddr 2.1.10-1 all
Python module for working with IP addresses, both IPv4 and IPv6
ii python-jinja2 2.6-1 amd64
small but fast and easy to use stand-alone template engine
ii python-markupsafe 0.15-1 amd64
XML/HTML/XHTML Markup safe string for Python
ii python-minimal 2.7.3-4+deb7u1 all
minimal subset of the Python language (default version)
ii python-openssl 0.13-2+deb7u1 amd64
Python 2 wrapper around the OpenSSL library
ii python-pam 0.4.2-13 amd64
Python interface to the PAM library
ii python-paramiko 1.7.7.1-3.1 all
Make ssh v2 connections with Python
ii python-parsley 1.2-1~bpo70+1 all
pattern-matching language based on OMeta and Python
ii python-pkg-resources 0.6.24-1 all
Package Discovery and Resource Access using pkg_resources
ii python-pyasn1 0.1.3-1 all
ASN.1 library for Python
ii python-pypcap 1.1.2+debian-2.2 amd64
object-oriented Python interface for libpcap
ii python-pyptlib 0.0.6-1~d70.wheezy+1 all
library for Tor pluggable transports written in Python
ii python-scapy 2.2.0-1 all
Packet generator/sniffer and network scanner/discovery
ii python-selinux 2.1.9-5 amd64
Python bindings to SELinux shared libraries
ii python-serial 2.5-2.1 all
pyserial - module encapsulating access for the serial port
ii python-sqlalchemy 0.7.8-1 all
SQL toolkit and Object Relational Mapper for Python
ii python-sqlalchemy-ext 0.7.8-1 amd64
SQL toolkit and Object Relational Mapper for Python - C extension
ii python-support 1.0.15 all
automated rebuilding support for Python modules
ii python-twisted 13.2.0-1~bpo70+1 all
Event-based framework for internet applications (dependency package)
ii python-twisted-bin 13.2.0-1~bpo70+1 amd64
Event-based framework for internet applications
ii python-twisted-conch 1:13.2.0-1~bpo70+1 all
Twisted SSH Implementation
ii python-twisted-core 13.2.0-1~bpo70+1 all
Event-based framework for internet applications
ii python-twisted-lore 13.2.0-1~bpo70+1 all
Documentation generator with HTML and LaTeX support
ii python-twisted-mail 13.2.0-1~bpo70+1 all
SMTP, IMAP and POP protocol implementation
ii python-twisted-names 13.2.0-1~bpo70+1 all
DNS protocol implementation with client and server
ii python-twisted-news 13.2.0-1~bpo70+1 all
NNTP protocol implementation with client and server
ii python-twisted-runner 13.2.0-1~bpo70+1 amd64
Process management, including an inetd server
ii python-twisted-web 13.2.0-1~bpo70+1 all
HTTP protocol implementation together with clients and servers
ii python-twisted-words 13.2.0-1~bpo70+1 all
Chat and Instant Messaging
ii python-txsocksx 1.13.0.3-1~bpo70+1 all
SOCKS{4,4a,5} endpoints for Twisted
ii python-txtorcon 0.11.0-1~bpo70+1 all
Twisted-based asynchronous Tor control protocol implementation
ii python-yaml 3.10-4 amd64
YAML parser and emitter for Python
ii python-zope.interface 3.6.1-3 amd64
Interfaces for Python
rc python2.6 2.6.8-1.1 amd64
Interactive high-level object-oriented language (version 2.6)
rc python2.6-minimal 2.6.8-1.1 amd64
Minimal subset of the Python language (version 2.6)
ii python2.7 2.7.3-6+deb7u2 amd64
Interactive high-level object-oriented language (version 2.7)
ii python2.7-dev 2.7.3-6+deb7u2 amd64
Header files and a static library for Python (v2.7)
ii python2.7-minimal 2.7.3-6+deb7u2 amd64
Minimal subset of the Python language (version 2.7)
}}}
{{{
ps axuwwww|head -1; ps axuwwww|egrep ^bwau.\*python
bwauth 4567 0.0 0.2 34252 2940 ? SN Jan23 0:00 python
./bwauthority.py ./data/scanner.1/bwauthority.cfg
bwauth 4568 0.0 0.2 34252 2932 ? SN Jan23 0:00 python
./bwauthority.py ./data/scanner.2/bwauthority.cfg
bwauth 4569 0.0 0.2 34252 2940 ? SN Jan23 0:00 python
./bwauthority.py ./data/scanner.3/bwauthority.cfg
bwauth 4570 0.0 0.2 34252 2928 ? SN Jan23 0:00 python
./bwauthority.py ./data/scanner.4/bwauthority.cfg
bwauth 24794 12.4 17.8 311752 183176 ? SNl 19:49 14:57 python
bwauthority_child.py ./data/scanner.3/bwauthority.cfg 4
bwauth 24819 14.2 17.8 311788 183204 ? SNl 19:52 16:38 python
bwauthority_child.py ./data/scanner.4/bwauthority.cfg 12
bwauth 24828 12.3 17.8 311940 183680 ? SNl 19:54 14:13 python
bwauthority_child.py ./data/scanner.2/bwauthority.cfg 17
bwauth 24833 10.4 18.3 316632 188580 ? SNl 19:55 12:01 python
bwauthority_child.py ./data/scanner.1/bwauthority.cfg 6
}}}
{{{
NOTE: this is a tweaked bwauthority.cfg, max_fetch_time = 30
grep "Timer exceeded limit: " -r data/scanner.1 | grep "Feb 03" | wc -l
410
}}}
{{{
grep "Launching stream request for url" -r data/scanner.1 | grep "Feb 03"
| wc -l
3728
}}}
Sample bwauthority.cfg
_pct ranges are: 0-30, 30-60, 60-80, 80-100
I set circs_per_node to 3 because I was in a hurry. May not be ideal.
{{{
[TorCtl]
loglevel=WARN
tor_host = 127.0.0.1
tor_port = 9110
control_host = 127.0.0.1
control_port = 9111
control_pass =
# XXX: Unused
meta_host = 127.0.0.1
meta_port = 9112
[BwAuthority]
out_dir = ./data/scanner.1/scan-data
pid_file = ./data/scanner.1/bwauthority.pid
db_url =
tor_dir = ./data/tor
start_pct = 0
stop_pct = 30
save_every = 0
nodes_per_slice = 100
circs_per_node = 3
min_streams = 1
max_fetch_time = 30
sleep_start = 01:30
sleep_stop = 01:30
}}}
Memory Info
{{{
KiB Mem: 1027068 total, 957124 used, 69944 free, 4996 buffers
KiB Swap: 2047996 total, 95808 used, 1952188 free, 69540 cached
}}}
Single core VM
{{{
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 1.4.0
stepping : 3
microcode : 0x1000065
cpu MHz : 2799.998
cache size : 512 KB
}}}
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/14177#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list