tor-commits September 2012

tor-commits@lists.torproject.org

18 participants
792 discussions

[tor/maint-0.2.3] Fix man page typo
by arma＠torproject.org 14 Sep '12

14 Sep '12

commit 909691f1ae9e2b8cd806cdf43b43f99b97c5c0f4 Author: Robert Ransom <rransom.8774(a)gmail.com> Date: Thu Sep 13 23:25:03 2012 -0700 Fix man page typo --- doc/tor.1.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/doc/tor.1.txt b/doc/tor.1.txt index 0bd9709..5aac054 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -81,7 +81,7 @@ COMMAND-LINE OPTIONS Other options can be specified on the command-line in the format "--option value", in the format "option value", or in a configuration file. For instance, you can tell Tor to start listening for SOCKS connections on port -9999 by passing --SOCKSPort 9999 or SOCKPort 9999 to it on the command line, +9999 by passing --SOCKSPort 9999 or SOCKSPort 9999 to it on the command line, or by putting "SOCKSPort 9999" in the configuration file. You will need to quote options with spaces in them: if you want Tor to log all debugging messages to debug.log, you will probably need to say --Log 'debug file

1 0

[ooni-probe/master] Fixed imports.
by isis＠torproject.org 14 Sep '12

14 Sep '12

commit cf5e13c241389f8192a587c0601a8cf561329ae7 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Sep 14 02:20:52 2012 +0000 Fixed imports. --- ooni/lib/Makefile | 30 ++ ooni/lib/__init__.py | 43 +--- ooni/lib/txscapy | 1 - ooni/lib/txscapy.py | 348 ++++++++++++++++++++ ooni/lib/txtraceroute | 1 - ooni/lib/txtraceroute.py | 752 ++++++++++++++++++++++++++++++++++++++++++++ ooni/ooniprobe.py | 10 +- ooni/plugins/blocking.py | 2 +- ooni/plugins/dnstamper.py | 6 +- ooni/plugins/httphost.py | 2 +- ooni/plugins/tcpconnect.py | 5 +- ooni/plugoo/nodes.py | 6 +- ooni/plugoo/reports.py | 2 +- ooni/plugoo/tests.py | 3 +- ooni/utils/log.py | 1 - 15 files changed, 1152 insertions(+), 60 deletions(-) diff --git a/ooni/lib/Makefile b/ooni/lib/Makefile new file mode 100644 index 0000000..3b0c922 --- /dev/null +++ b/ooni/lib/Makefile @@ -0,0 +1,30 @@ +all: txtorcon txtraceroute + +txtraceroute: + echo "Processing dependency txtraceroute..." + git clone https://github.com/hellais/txtraceroute.git txtraceroute.git + mv txtraceroute.git/txtraceroute.py txtraceroute.py + rm -rf txtraceroute.git + +txtorcon: + echo "Processing dependency txtorcon..." + git clone https://github.com/meejah/txtorcon.git txtorcon.git + mv txtorcon.git/txtorcon txtorcon + rm -rf txtorcon.git + +clean: + rm -rf txtorcon + rm -rf txtraceroute.py + +#txscapy: +# echo "Processing dependency txscapy" +# git clone https://github.com/hellais/txscapy.git txscapy.git +# mv txscapy.git/txscapy.py txscapy.py +# rm -rf txscapy.git + +#rfc3339: +# echo "Processing RFC3339 dependency" +# hg clone https://bitbucket.org/henry/rfc3339 rfc3339 +# mv rfc3339/rfc3339.py rfc3339.py +# rm -rf rfc3339 + diff --git a/ooni/lib/__init__.py b/ooni/lib/__init__.py index 0fd36c5..611d50c 100644 --- a/ooni/lib/__init__.py +++ b/ooni/lib/__init__.py @@ -1,40 +1,5 @@ -import pkgutil -import sys -from os import listdir, path +from sys import path as syspath +from os import path as ospath -__all__ = ['txtorcon', 'txscapy', 'txtraceroute'] - -__sub_modules__ = [ ] - -def callback(arg, directory, files): - for file in listdir(directory): - fullpath = path.abspath(file) - if path.isdir(fullpath) and not path.islink(fullpath): - __sub_modules__.append(fullpath) - sys.path.append(fullpath) - -path.walk(".", callback, None) - -def load_submodules(init, list): - for subdir in list: - contents=[x for x in pkgutil.iter_modules(path=subdir, - prefix='ooni.lib.')] - for loader, module_name, ispkg in contents: - init_dot_module = init + "." + module_name - if init_dot_module in sys.modules: - module = sys.modules[module_name] - else: - if module_name in __all__: - grep = loader.find_module(module_name) - module = grep.load_module(module_name) - else: - module = None - - if module is not None: - globals()[module_name] = module - -load_submodules(__name__, __sub_modules__) - -print "system paths are: %s" % sys.path -print "globals are: %s" % globals() -print "system modules are: %s" % sys.modules +pwd = ospath.dirname(__file__) +syspath.append(pwd) diff --git a/ooni/lib/txscapy b/ooni/lib/txscapy deleted file mode 160000 index 19fb281..0000000 --- a/ooni/lib/txscapy +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 19fb28150c0b31f16a1ae2bc0aadeb6fd3c259bf diff --git a/ooni/lib/txscapy.py b/ooni/lib/txscapy.py new file mode 100644 index 0000000..4d83dce --- /dev/null +++ b/ooni/lib/txscapy.py @@ -0,0 +1,348 @@ +# -*- coding:utf8 -*- +""" + txscapy + ****** + (c) 2012 Arturo Filastò + a twisted wrapper for scapys send and receive functions. + + This software has been written to be part of OONI, the Open Observatory of + Network Interference. More information on that here: http://ooni.nu/ + +""" + +import struct +import socket +import os +import sys +import time + +from twisted.internet import protocol, base, fdesc, error, defer +from twisted.internet import reactor, threads +from twisted.python import log +from zope.interface import implements + +from scapy.all import Gen +from scapy.all import SetGen + +LINUX=sys.platform.startswith("linux") +OPENBSD=sys.platform.startswith("openbsd") +FREEBSD=sys.platform.startswith("freebsd") +NETBSD=sys.platform.startswith("netbsd") +DARWIN=sys.platform.startswith("darwin") +SOLARIS=sys.platform.startswith("sunos") +WINDOWS=sys.platform.startswith("win32") + +from scapy.all import RawPcapWriter, MTU, BasePacketList, conf +class PcapWriter(RawPcapWriter): + def __init__(self, filename, linktype=None, gz=False, endianness="", + append=False, sync=False): + RawPcapWriter.__init__(self, filename, linktype=None, gz=False, + endianness="", append=False, sync=False) + fdesc.setNonBlocking(self.f) + + def _write_header(self, pkt): + if self.linktype == None: + if type(pkt) is list or type(pkt) is tuple or isinstance(pkt, BasePacketList): + pkt = pkt[0] + try: + self.linktype = conf.l2types[pkt.__class__] + except KeyError: + self.linktype = 1 + RawPcapWriter._write_header(self, pkt) + + def _write_packet(self, packet): + sec = int(packet.time) + usec = int(round((packet.time-sec)*1000000)) + s = str(packet) + caplen = len(s) + RawPcapWriter._write_packet(self, s, sec, usec, caplen, caplen) + +class ScapySocket(object): + MTU = 1500 + def __init__(self, filter=None, iface=None, nofilter=None): + from scapy.all import conf + self.ssocket = conf.L3socket(filter=filter, iface=iface, nofilter=nofilter) + + def fileno(self): + return self.ssocket.ins.fileno() + + def send(self, data): + return self.ssocket.send(data) + + def recv(self): + if FREEBSD or DARWIN: + return self.ssocket.nonblock_recv() + else: + return self.ssocket.recv(self.MTU) + +class Scapy(object): + """ + A twisted based wrapper for scapy send and receive functionality. + + It sends packets inside of a threadpool and receives packets using the + libdnet receive non blocking file descriptor. + """ + min = 2 + max = 6 + debug = True + write_only_answers = False + pcapwriter = None + recv = False + + def __init__(self, pkts=None, maxPacketSize=8192, reactor=None, filter=None, + iface=None, nofilter=None, pcapfile=None): + + if self.debug: + log.startLogging(sys.stdout) + + self.maxPacketSize = maxPacketSize + if not reactor: + from twisted.internet import reactor + + self._reactor = reactor + + if pkts: + self._buildPacketQueues(pkts) + self._buildSocket() + + self.cthreads = 0 + self.mthreads = 80 + + self.running = False + self.done = False + self.finished = False + + import thread + from twisted.python import threadpool + self.threadID = thread.get_ident + self.threadpool = threadpool.ThreadPool(self.min, self.max) + self.startID = self._reactor.callWhenRunning(self._start) + + self.deferred = defer.Deferred() + + if pcapfile: + self.pcapwriter = PcapWriter(pcapfile) + + def _buildSocket(self, filter=None, iface=None, nofilter=None): + self.socket = ScapySocket(filter, iface, nofilter) + if self.recv: + self._reactor.addReader(self) + + def _buildPacketQueues(self, pkts): + """ + Converts the list of packets to a Scapy generator and sets up all the + necessary attributes for understanding if all the needed responses have + been received. + """ + if not isinstance(pkts, Gen): + self.pkts = SetGen(pkts) + + self.outqueue = [p for p in pkts] + + self.total_count = len(self.outqueue) + self.answer_count = 0 + self.out_count = 0 + + self.hsent = {} + for p in self.outqueue: + h = p.hashret() + if h in self.hsent: + self.hsent[h].append(p) + else: + self.hsent[h] = [p] + + + def gotAnswer(self, answer, question): + """ + Got a packet that has been identified as an answer to one of the sent + out packets. + + If the answer count matches the sent count the finish callback is + fired. + + @param answer: the packet received on the wire. + + @param question: the sent packet that matches that response. + + """ + + if self.pcapwriter and self.write_only_answers: + self.pcapwriter.write(question) + self.pcapwriter.write(answer) + self.answer_count += 1 + if self.answer_count >= self.total_count: + print "Got all the answers I need" + self.deferred.callback(None) + + def processAnswer(self, pkt, hlst): + """ + Checks if the potential answer is in fact an answer to one of the + matched sent packets. Uses the scapy .answers() function to verify + this. + + @param pkt: The packet to be tested if is the answer to a sent packet. + + @param hlst: a list of packets that match the hash for an answer to + pkt. + """ + for i in range(len(hlst)): + if pkt.answers(hlst[i]): + self.gotAnswer(pkt, hlst[i]) + + def fileno(self): + """ + Returns a fileno for use by twisteds Reader. + """ + return self.socket.fileno() + + def processPacket(self, pkt): + """ + Override this method to process your packets. + + @param pkt: the packet that has been received. + """ + #pkt.show() + + + def doRead(self): + """ + There is something to be read on the wire. Do all the processing on the + received packet. + """ + pkt = self.socket.recv() + if self.pcapwriter and not self.write_only_answers: + self.pcapwriter.write(pkt) + self.processPacket(pkt) + + h = pkt.hashret() + if h in self.hsent: + hlst = self.hsent[h] + self.processAnswer(pkt, hlst) + + def logPrefix(self): + """ + The prefix to be prepended in logging. + """ + return "txScapy" + + def _start(self): + """ + Start the twisted thread pool. + """ + self.startID = None + return self.start() + + def start(self): + """ + Actually start the thread pool. + """ + if not self.running: + self.threadpool.start() + self.shutdownID = self._reactor.addSystemEventTrigger( + 'during', 'shutdown', self.finalClose) + self.running = True + + def sendPkt(self, pkt): + """ + Send a packet to the wire. + + @param pkt: The packet to be sent. + """ + self.socket.send(pkt) + + def sr(self, pkts, filter=None, iface=None, nofilter=0, *args, **kw): + """ + Wraps the scapy sr function. + + @param nofilter: put 1 to avoid use of bpf filters + + @param retry: if positive, how many times to resend unanswered packets + if negative, how many times to retry when no more packets are + answered (XXX to be implemented) + + @param timeout: how much time to wait after the last packet has + been sent (XXX to be implemented) + + @param multi: whether to accept multiple answers for the same + stimulus (XXX to be implemented) + + @param filter: provide a BPF filter + @param iface: listen answers only on the given interface + """ + self.recv = True + self._sendrcv(pkts, filter=filter, iface=iface, nofilter=nofilter) + + def send(self, pkts, filter=None, iface=None, nofilter=0, *args, **kw): + """ + Wraps the scapy send function. Its the same as send and receive, except + it does not receive. Who would have ever guessed? ;) + + @param nofilter: put 1 to avoid use of bpf filters + + @param retry: if positive, how many times to resend unanswered packets + if negative, how many times to retry when no more packets are + answered (XXX to be implemented) + + @param timeout: how much time to wait after the last packet has + been sent (XXX to be implemented) + + @param multi: whether to accept multiple answers for the same + stimulus (XXX to be implemented) + + @param filter: provide a BPF filter + @param iface: listen answers only on the given interface + """ + self.recv = False + self._sendrcv(pkts, filter=filter, iface=iface, nofilter=nofilter) + + def _sendrcv(self, pkts, filter=None, iface=None, nofilter=0): + self._buildSocket(filter, iface, nofilter) + self._buildPacketQueues(pkts) + def sent(cb): + if self.cthreads < self.mthreads and not self.done: + pkt = None + try: + pkt = self.outqueue.pop() + except: + self.done = True + if not self.recv: + self.deferred.callback(None) + return + d = threads.deferToThreadPool(reactor, self.threadpool, + self.sendPkt, pkt) + d.addCallback(sent) + return d + + for x in range(self.mthreads): + try: + pkt = self.outqueue.pop() + except: + self.done = True + return + if self.cthreads >= self.mthreads and self.done: + return + d = threads.deferToThreadPool(reactor, self.threadpool, + self.sendPkt, pkt) + d.addCallback(sent) + return d + + def connectionLost(self, why): + pass + + def finalClose(self): + """ + Clean all the thread related stuff up. + """ + self.shutdownID = None + self.threadpool.stop() + self.running = False + +def txsr(*args, **kw): + tr = Scapy(*args, **kw) + tr.sr(*args, **kw) + return tr.deferred + +def txsend(*arg, **kw): + tr = Scapy(*arg, **kw) + tr.send(*arg, **kw) + return tr.deferred diff --git a/ooni/lib/txtraceroute b/ooni/lib/txtraceroute deleted file mode 160000 index 067a260..0000000 --- a/ooni/lib/txtraceroute +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 067a2609390e77bf9187275638cf8786efef7e13 diff --git a/ooni/lib/txtraceroute.py b/ooni/lib/txtraceroute.py new file mode 100644 index 0000000..8182b34 --- /dev/null +++ b/ooni/lib/txtraceroute.py @@ -0,0 +1,752 @@ +#!/usr/bin/env python +# coding: utf-8 +# +# Copyright (c) 2012 Alexandre Fiori +# Arturo Filastò +# +# Permission is hereby granted, free of charge, to any person obtaining +# a copy of this software and associated documentation files (the +# "Software"), to deal in the Software without restriction, including +# without limitation the rights to use, copy, modify, merge, publish, +# distribute, sublicense, and/or sell copies of the Software, and to +# permit persons to whom the Software is furnished to do so, subject to +# the following conditions: +# +# The above copyright notice and this permission notice shall be +# included in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +import json +import operator +import os +import socket +import struct +import sys +import time +import random +import itertools +from pprint import pprint + +from twisted.internet import defer +from twisted.internet import reactor +from twisted.internet import threads +from twisted.python import usage +from twisted.web.client import getPage + +class iphdr(object): + """ + This represents an IP packet header. + + XXX enable IP_TIMESTAMP in setsockopt + to get the timestamp of when the router says it has gotten an ICMP + timeout. + + @assemble packages the packet + @disassemble disassembles the packet + """ + def __init__(self, proto=socket.IPPROTO_ICMP, src="0.0.0.0", dst=None): + self.version = 4 + self.hlen = 5 + self.tos = 0 + self.length = 20 + self.id = os.getpid() + self.frag = 0 + self.ttl = 255 + self.proto = proto + self.cksum = 0 + self.src = src + self.saddr = socket.inet_aton(src) + self.dst = dst or "0.0.0.0" + self.daddr = socket.inet_aton(self.dst) + self.data = "" + + def assemble(self): + header = struct.pack('BBHHHBB', + (self.version & 0x0f) << 4 | (self.hlen & 0x0f), + self.tos, self.length + len(self.data), + socket.htons(self.id), self.frag, + self.ttl, self.proto) + self._raw = header + "\x00\x00" + self.saddr + self.daddr + self.data + return self._raw + + @classmethod + def disassemble(self, data): + self._raw = data + ip = iphdr() + pkt = struct.unpack('!BBHHHBBH', data[:12]) + ip.version = (pkt[0] >> 4 & 0x0f) + ip.hlen = (pkt[0] & 0x0f) + ip.tos, ip.length, ip.id, ip.frag, ip.ttl, ip.proto, ip.cksum = pkt[1:] + ip.saddr = data[12:16] + ip.daddr = data[16:20] + ip.src = socket.inet_ntoa(ip.saddr) + ip.dst = socket.inet_ntoa(ip.daddr) + return ip + + def __repr__(self): + return "IP (tos %s, ttl %s, id %s, frag %s, proto %s, length %s) " \ + "%s -> %s" % \ + (self.tos, self.ttl, self.id, self.frag, self.proto, + self.length, self.src, self.dst) + +class tcphdr(object): + def __init__(self, data="", sport=4242, dport=4242): + self.seq = 0 + self.hlen = 44 + self.flags = 2 + self.wsize = 200 + self.cksum = 123 + self.options = 0 + self.mss = 1460 + self.dport = dport + self.sport = sport + + def assemble(self): + header = struct.pack("!HHL", self.sport, self.dport, self.seq) + header += '\x00\x00\x00\x00' + header += struct.pack("!HHH", (self.hlen & 0xff) << 10 | (self.flags & + 0xff), self.wsize, self.cksum) + header += "\x00\x00" + options = '\x02\x04\x05\xb4\x01\x03\x03\x01\x01\x01\x08\x0a' + options += '\x4d\xcf\x52\x33\x00\x00\x00\x00\x04\x02\x00\x00' + # XXX There is something wrong here fixme + # options = struct.pack("!LBBBBBB", self.mss, 1, 3, 3, 1, 1, 1) + # options += struct.pack("!BBL", 8, 10, 1209452188) + # options += '\00'*4 + # options += struct.pack("!BB", 4, 2) + # options += '\00' + self._raw = header+options + return self._raw + + @classmethod + def checksum(self, data): + pass + + def __repr__(self): + return "<TCPPacket (sport: %s dport: %s seq: %s) " %\ + (self.sport, self.dport, self.seq) + + @classmethod + def disassemble(self, data): + self._raw = data + tcp = tcphdr() + pkt = struct.unpack("!HHL", data[:8]) + tcp.sport, tcp.dport, tcp.seq = pkt + if len(data) > 10: + pkt = struct.unpack("!H", data[8:10]) + tcp.hlen = (pkt[0] >> 10 ) & 0xff + tcp.flags = pkt[0] & 0xff + tcp.wsize, tcp.cksum = struct.unpack("!HH", data[20:24]) + return tcp + +class udphdr(object): + def __init__(self, data="", sport=4242, dport=4242): + self.dport = dport + self.sport = sport + self.cksum = 0 + self.length = 0 + self.data = data + + def assemble(self): + self.length = len(self.data) + 8 + part1 = struct.pack("!HHH", self.sport, self.dport, self.length) + cksum = self.checksum(self.data) + cksum = struct.pack("!H", cksum) + + self._raw = part1 + cksum + self.data + return self._raw + + @classmethod + def checksum(self, data): + # XXX implement proper checksum + cksum = 0 + return cksum + + def __repr__(self): + return "<UDPPacket (sport %s, dport %s, length %s, data %s)>" % \ + (self.sport, self.dport, self.length, self.data) + + @classmethod + def disassemble(self, data): + self._raw = data + udp = udphdr() + pkt = struct.unpack("!HHHH", data[:8]) + udp.sport, udp.dport, udp.length, udp.cksum = pkt + udp.data = data[8:] + return udp + +class icmphdr(object): + def __init__(self, data=""): + self.type = 8 + self.code = 0 + self.cksum = 0 + self.id = os.getpid() + self.sequence = 0 + self.data = data + + def assemble(self): + part1 = struct.pack("BB", self.type, self.code) + part2 = struct.pack("!HH", self.id, self.sequence) + cksum = self.checksum(part1 + "\x00\x00" + part2 + self.data) + cksum = struct.pack("!H", cksum) + self._raw = part1 + cksum + part2 + self.data + return self._raw + + @classmethod + def checksum(self, data): + if len(data) & 1: + data += "\x00" + cksum = reduce(operator.add, + struct.unpack('!%dH' % (len(data) >> 1), data)) + cksum = (cksum >> 16) + (cksum & 0xffff) + cksum += (cksum >> 16) + cksum = (cksum & 0xffff) ^ 0xffff + return cksum + + @classmethod + def disassemble(self, data): + self._raw = data + icmp = icmphdr() + pkt = struct.unpack("!BBHHH", data) + icmp.type, icmp.code, icmp.cksum, icmp.id, icmp.sequence = pkt + return icmp + + def __repr__(self): + return "ICMP (type %s, code %s, id %s, sequence %s)" % \ + (self.type, self.code, self.id, self.sequence) + + +def pprintp(packet): + """ + Used to pretty print packets. + """ + lines = [] + line = [] + for i, byte in enumerate(packet): + line.append(("%.2x" % ord(byte), byte)) + if (i + 1) % 8 == 0: + lines.append(line) + line = [] + + lines.append(line) + + for row in lines: + left = "" + right = " " * (8 - len(row)) + for y in row: + left += "%s " % y[0] + right += "%s" % y[1] + + print left + " " + right + +(a)defer.inlineCallbacks +def geoip_lookup(ip): + try: + r = yield getPage("http://freegeoip.net/json/%s" % ip) + d = json.loads(r) + items = [d["country_name"], d["region_name"], d["city"]] + text = ", ".join([s for s in items if s]) + defer.returnValue(text.encode("utf-8")) + except Exception: + defer.returnValue("Unknown location") + + +(a)defer.inlineCallbacks +def reverse_lookup(ip): + try: + r = yield threads.deferToThread(socket.gethostbyaddr, ip) + defer.returnValue(r[0]) + except Exception: + defer.returnValue(None) + + +class Hop(object): + def __init__(self, target, ttl, proto, sport=None, dport=None): + self.proto = proto + self.dport = dport + self.sport = sport + + self.found = False + self.tries = 0 + self.last_try = 0 + self.remote_ip = None + self.remote_icmp = None + self.remote_host = None + self.location = "" + + self.ttl = ttl + self.ip = iphdr(dst=target) + self.ip.ttl = ttl + self.ip.id += ttl + if self.proto == "icmp": + self.icmp = icmphdr('\x00'*20) + self.icmp.id = self.ip.id + self.ip.data = self.icmp.assemble() + elif self.proto == "udp": + self.udp = udphdr('\x00'*20, self.sport, self.dport) + self.ip.data = self.udp.assemble() + self.ip.proto = socket.IPPROTO_UDP + else: + self.tcp = tcphdr('\x42'*20, self.sport, self.dport) + self.ip.data = self.tcp.assemble() + self.ip.proto = socket.IPPROTO_TCP + + self._pkt = self.ip.assemble() + + @property + def pkt(self): + self.tries += 1 + self.last_try = time.time() + return self._pkt + + def get(self): + if self.found: + if self.remote_host: + ip = self.remote_host + else: + ip = self.remote_ip.src + ping = self.found - self.last_try + else: + ip = None + ping = None + + location = self.location if self.location else None + return {'ttl': self.ttl, 'ping': ping, 'ip': ip, 'location': location, + 'proto': self.proto, 'dport': self.dport, 'sport': self.sport} + + def __repr__(self): + if self.found: + if self.remote_host: + ip = ":: %s" % self.remote_host + else: + ip = ":: %s" % self.remote_ip.src + ping = "%0.3fs" % (self.found - self.last_try) + else: + ip = "??" + ping = "-" + + location = ":: %s" % self.location if self.location else "" + return "%02d. %s %s %s (%s, sport: %s dport: %s)" % (self.ttl, ping, ip, location, self.proto, self.sport, self.dport) + +class TracerouteResult(object): + """ + Used to store the results of a Traceroute. + """ + #src_ports = [0, 9090] + #dst_ports = [0, 21, 123, 80, 443] + src_ports = [0, 80] + dst_ports = [0, 80] + hops = [] + done = False + + def __init__(self, protocol): + self.protocol = protocol + self.probes = {} + + if protocol == "icmp": + self.current = None + else: + self.current = {} + for src, dst in itertools.product(self.src_ports, + self.dst_ports): + if src not in self.probes: + self.probes[src] = {} + self.probes[src][dst] = [] + + if src not in self.current: + self.current[src] = {} + self.current[src][dst] = None + + def get_current_probes(self): + if self.protocol == "icmp": + return self.current + + def add_to_current_probes(self, probe): + if self.protocol == "icmp": + self.current = probe + else: + self.current[probe.sport][probe.dport] = probe + + def is_in_progress(self): + if self.protocol == "icmp": + progress = self.current + else: + progress = None + for x in self.current: + for y in self.current[x]: + if self.current[x][y] != None: + progress = True + if progress is None: + return False + else: + return True + + def get(self, src=None, dst=None): + if self.protocol == "icmp": + return self.probes + else: + return self.probes[src][dst] + + def append(self, probe, src=None, dst=None): + if self.protocol == "icmp": + self.probes.append(hop) + else: + self.probes[src][dst].append(probe) + + def pop(self, src=None, dst=None): + if self.protocol == "icmp": + hop = self.current + self.current = None + return hop + + elif (dst != None) and (src != None): + hop = self.current[src][dst] + self.current[src][dst] = None + return hop + + else: + raise Exception("Did not specify dst and src ports") + + @classmethod + def hops(self, target, ttl): + """ + Generates a set of ooni probes for traceroute based network tampering + detection. + + We send in one round a set of packets with same TTL but on all protocols + and with all possible source and destination ports. + """ + hops = [] + for src, dst in itertools.product(self.src_ports, self.dst_ports): + hops.append(Hop(target, ttl, + "tcp", src, dst)) + hops.append(Hop(target, ttl, + "udp", src, dst)) + hops.append(Hop(target, ttl, "icmp", 0, 0)) + return hops + +class TracerouteProtocol(object): + def __init__(self, target, **settings): + + self.target = target + self.settings = settings + self.verbose = settings.get("verbose") + self.proto = settings.get("proto") + self.rfd = socket.socket(socket.AF_INET, socket.SOCK_RAW, + socket.IPPROTO_ICMP) + self.sfd = {} + + # Create the data structures to contain the test results + self.traceroute = {} + self.traceroute["tcp"] = TracerouteResult("tcp") + self.traceroute["udp"] = TracerouteResult("udp") + self.traceroute["icmp"] = TracerouteResult("icmp") + + if self.settings.get("ooni"): + self.sfd["tcp"] = socket.socket(socket.AF_INET, socket.SOCK_RAW, + socket.IPPROTO_TCP) + self.sfd["icmp"] = socket.socket(socket.AF_INET, socket.SOCK_RAW, + socket.IPPROTO_ICMP) + self.sfd["udp"] = socket.socket(socket.AF_INET, socket.SOCK_RAW, + socket.IPPROTO_UDP) + elif self.proto == "icmp": + self.sfd["icmp"] = socket.socket(socket.AF_INET, socket.SOCK_RAW, + socket.IPPROTO_ICMP) + elif self.proto == "udp": + self.sfd["udp"] = socket.socket(socket.AF_INET, socket.SOCK_RAW, + socket.IPPROTO_UDP) + elif self.proto == "tcp": + self.sfd["tcp"] = socket.socket(socket.AF_INET, socket.SOCK_RAW, + socket.IPPROTO_TCP) + + # Let me add IP Headers myself, just give me a socket! + self.rfd.setsockopt(socket.IPPROTO_IP, socket.IP_HDRINCL, 1) + for fd in self.sfd: + self.sfd[fd].setsockopt(socket.IPPROTO_IP, socket.IP_HDRINCL, 1) + + self.out_queue = [] + self.waiting = True + self.deferred = defer.Deferred() + + reactor.addReader(self) + reactor.addWriter(self) + + # send 1st probe packet(s) + if self.settings.get("ooni"): + hops = list(TracerouteResult.hops(self.target, 1)) + else: + hops = [Hop(self.target, 1, + settings.get("proto"), + self.settings.get("sport"), + self.settings.get("dport"))] + for hop in hops: + # Store the to be completed items inside of a dictionary + self.traceroute[hop.proto].add_to_current_probes(hop) + self.out_queue.append(hop) + + def logPrefix(self): + return "TracerouteProtocol(%s)" % self.target + + def fileno(self): + return self.rfd.fileno() + + @defer.inlineCallbacks + def hopFound(self, hop, ip, icmp, ref, subref): + hop.remote_ip = ip + hop.remote_icmp = icmp + + if (ip and icmp): + hop.found = time.time() + if self.settings.get("geoip_lookup") is True: + hop.location = yield geoip_lookup(ip.src) + + if self.settings.get("reverse_lookup") is True: + hop.remote_host = yield reverse_lookup(ip.src) + + ttl = hop.ttl + 1 + + if (ttl > (self.settings.get("max_hops", 30) + 1)): + done = True + else: + done = False + + if not done: + cb = self.settings.get("hop_callback") + if callable(cb): + yield defer.maybeDeferred(cb, hop) + + if not self.waiting: + if self.deferred: + self.deferred.callback(self.hops) + self.deferred = None + else: + hops = [] + if self.settings.get("ooni"): + if not (self.traceroute["icmp"].is_in_progress() or + self.traceroute["tcp"].is_in_progress() or + self.traceroute["udp"].is_in_progress()): + # Add hops only if we are not in progress + hops = list(TracerouteResult.hops(self.target, ttl)) + else: + hops = [Hop(self.target, ttl, + settings.get("proto"), + self.settings.get("sport"), + self.settings.get("dport"))] + + for hop in hops: + # Store the to be completed items inside of a dictionary + self.traceroute[hop.proto].add_to_current_probes(hop) + self.out_queue.append(hop) + + def doRead(self): + if not self.waiting: + return + + pkt = self.rfd.recv(4096) + # disassemble ip header + ip = iphdr.disassemble(pkt[:20]) + + if self.verbose: + print "Got this packet:" + print "src %s" % ip.src + pprintp(pkt) + + # Not interested in non ICMP packets. + if ip.proto != socket.IPPROTO_ICMP: + return + + found = False + foundHop = None + + # disassemble icmp header + icmp = icmphdr.disassemble(pkt[20:28]) + + if self.verbose: + print icmp + + # If it's an ICMP Echo reply then our ICMP probe has hit destination + if icmp.type == 0 and icmp.id == self.current_hop["icmp"][1].icmp.id: + foundHop = self.traceroute["icmp"].pop() + found = True + + elif icmp.type == 11: + # disassemble referenced ip header + ref = iphdr.disassemble(pkt[28:48]) + subref = None + + if self.verbose: + print ref + + if ref.dst == self.target: + found = True + + if ref.proto == socket.IPPROTO_UDP: + subref = udphdr.disassemble(pkt[48:]) + proto = "udp" + + elif ref.proto == socket.IPPROTO_TCP: + subref = tcphdr.disassemble(pkt[48:]) + proto = "tcp" + + else: + proto = "icmp" + + if subref: + sport = subref.sport + dport = subref.dport + else: + sport = None + dport = None + # Remove completed hops + foundHop = self.traceroute[proto].pop(sport, + dport) + + if ip.src == self.target: + self.waiting = False + + if found: + self.hopFound(foundHop, ip, icmp, ref, subref) + elif foundHop: + self.hopFound(foundHop, ip, icmp, ref, subref) + + def hopTimeout(self, hop): + if not hop.found: + if hop.tries < self.settings.get("max_tries", 3): + # retry + hop.tries += 1 + self.out_queue.append(hop) + else: + # give up and move forward + self.traceroute[hop.proto].pop(hop.dport, + hop.sport) + self.hopFound(hop, None, None, None, None) + + def doWrite(self): + if self.waiting and self.out_queue: + hop = self.out_queue.pop(0) + pkt = hop.pkt + if self.verbose: + print "Sending this packet:" + pprintp(pkt) + print hop + + self.sfd[hop.proto].sendto(pkt, (hop.ip.dst, 0)) + + self.traceroute[hop.proto].add_to_current_probes(hop) + + timeout = self.settings.get("timeout", 1) + reactor.callLater(timeout, self.hopTimeout, hop) + + def connectionLost(self, why): + pass + + +def traceroute(target, **settings): + tr = TracerouteProtocol(target, **settings) + return tr.deferred + + +(a)defer.inlineCallbacks +def start_trace(target, **settings): + hops = yield traceroute(target, **settings) + if settings["hop_callback"] is None: + for hop in hops: + print hop + reactor.stop() + +class Options(usage.Options): + optFlags = [ + ["quiet", "q", "Only print results at the end."], + ["no-dns", "n", "Show numeric IPs only, not their host names."], + ["no-geoip", "g", "Do not collect and show GeoIP information"], + ["verbose", "v", "Be more verbose"], + ["ooni", "o", "Run the ooni common port multiprotocol traceroute"], + ["help", "h", "Show this help"], + ] + optParameters = [ + ["timeout", "t", 2, "Timeout for probe packets"], + ["tries", "r", 3, "How many tries before give up probing a hop"], + ["proto", "p", "icmp", "What protocol to use (tcp, udp, icmp)"], + ["dport", "d", random.randint(2**10, 2**16), "Destination port (TCP and UDP only)"], + ["sport", "s", random.randint(2**10, 2**16), "Source port (TCP and UDP only)"], + ["max_hops", "m", 30, "Max number of hops to probe"] + ] + +def main(): + def show(hop): + print hop + + defaults = dict(hop_callback=show, + reverse_lookup=True, + geoip_lookup=True, + timeout=2, + proto="icmp", + dport=None, + sport=None, + verbose=False, + ooni=False, + max_tries=3, + max_hops=30) + + if len(sys.argv) < 2: + print("Usage: %s [options] host" % (sys.argv[0])) + print("%s: Try --help for usage details." % (sys.argv[0])) + sys.exit(1) + + target = sys.argv.pop(-1) if sys.argv[-1][0] != "-" else "" + config = Options() + try: + config.parseOptions() + if not target: + raise + except usage.UsageError, e: + print("%s: %s" % (sys.argv[0], e)) + print("%s: Try --help for usage details." % (sys.argv[0])) + sys.exit(1) + + settings = defaults.copy() + if config.get("silent"): + settings["hop_callback"] = None + if config.get("no-dns"): + settings["reverse_lookup"] = False + if config.get("no-geoip"): + settings["geoip_lookup"] = False + if config.get("verbose"): + settings["verbose"] = True + if config.get("ooni"): + settings["ooni"] = True + if "timeout" in config: + settings["timeout"] = config["timeout"] + if "tries" in config: + settings["max_tries"] = config["tries"] + if "proto" in config: + settings["proto"] = config["proto"] + if "max_hops" in config: + settings["max_hops"] = config["max_hops"] + if "dport" in config: + settings["dport"] = int(config["dport"]) + if "sport" in config: + settings["sport"] = int(config["sport"]) + + if os.getuid() != 0: + print("traceroute needs root privileges for the raw socket") + sys.exit(1) + try: + target = socket.gethostbyname(target) + except Exception, e: + print("could not resolve '%s': %s" % (target, str(e))) + sys.exit(1) + + reactor.callWhenRunning(start_trace, target, **settings) + reactor.run() + +if __name__ == "__main__": + main() + diff --git a/ooni/ooniprobe.py b/ooni/ooniprobe.py index 95b5a18..539c2ac 100755 --- a/ooni/ooniprobe.py +++ b/ooni/ooniprobe.py @@ -27,9 +27,11 @@ from zope.interface.verify import verifyObject from zope.interface.exceptions import BrokenImplementation from zope.interface.exceptions import BrokenMethodImplementation -from ooni.plugoo import tests, work, assets, reports -from ooni.logo import getlogo -from ooni import plugins, log +from plugoo import tests, work, assets, reports +from plugoo.interface import ITest +from utils.logo import getlogo +from utils import log +import plugins __version__ = "0.0.1-prealpha" @@ -38,7 +40,7 @@ def retrieve_plugoo(): Get all the plugins that implement the ITest interface and get the data associated to them into a dict. """ - interface = tests.ITest + interface = ITest d = {} error = False for p in getPlugins(interface, plugins): diff --git a/ooni/plugins/blocking.py b/ooni/plugins/blocking.py index 72fd49f..f3c20e1 100644 --- a/ooni/plugins/blocking.py +++ b/ooni/plugins/blocking.py @@ -3,7 +3,7 @@ from twisted.python import usage from twisted.plugin import IPlugin from plugoo.assets import Asset -from plugoo.tests import ITest, TwistedTest +from plugoo.tests import ITest, OONITest class BlockingArgs(usage.Options): optParameters = [['asset', 'a', None, 'Asset file'], diff --git a/ooni/plugins/dnstamper.py b/ooni/plugins/dnstamper.py index 54cfbdf..3d373ce 100644 --- a/ooni/plugins/dnstamper.py +++ b/ooni/plugins/dnstamper.py @@ -38,9 +38,9 @@ from twisted.python import usage from twisted.plugin import IPlugin from zope.interface import implements -from ooni.plugoo.assets import Asset -from ooni.plugoo.tests import ITest, OONITest -from ooni import log +from plugoo.assets import Asset +from plugoo.tests import ITest, OONITest +from utils import log class AlexaAsset(Asset): """ diff --git a/ooni/plugins/httphost.py b/ooni/plugins/httphost.py index e8808bd..7c783a1 100644 --- a/ooni/plugins/httphost.py +++ b/ooni/plugins/httphost.py @@ -21,7 +21,7 @@ from twisted.python import usage from twisted.plugin import IPlugin from plugoo.assets import Asset -from plugoo.tests import ITest, TwistedTest +from plugoo.tests import ITest, OONITest class HTTPHostArgs(usage.Options): optParameters = [['asset', 'a', None, 'Asset file'], diff --git a/ooni/plugins/tcpconnect.py b/ooni/plugins/tcpconnect.py index db3d969..bbf62a5 100644 --- a/ooni/plugins/tcpconnect.py +++ b/ooni/plugins/tcpconnect.py @@ -9,8 +9,9 @@ from twisted.plugin import IPlugin from twisted.internet.protocol import Factory, Protocol from twisted.internet.endpoints import TCP4ClientEndpoint -from ooni.plugoo.tests import ITest, OONITest -from ooni.plugoo.assets import Asset +from plugoo.interface import ITest +from plugoo.tests import OONITest +from plugoo.assets import Asset from ooni.utils import log class tcpconnectArgs(usage.Options): diff --git a/ooni/plugoo/nodes.py b/ooni/plugoo/nodes.py index 0d01348..155f183 100644 --- a/ooni/plugoo/nodes.py +++ b/ooni/plugoo/nodes.py @@ -7,7 +7,7 @@ This contains all the code related to Nodes both network and code execution. - :copyright: (c) 2012 by Arturo Filastò. + :copyright: (c) 2012 by Arturo Filastò, Isis Lovecruft :license: see LICENSE for more details. """ @@ -20,10 +20,6 @@ try: except: print "Error: module paramiko is not installed." from pprint import pprint -try: - import pyXMLRPCssh -except: - print "Error: module pyXMLRPCssh is not installed." import sys import socks import xmlrpclib diff --git a/ooni/plugoo/reports.py b/ooni/plugoo/reports.py index ef15a04..d0d9af3 100644 --- a/ooni/plugoo/reports.py +++ b/ooni/plugoo/reports.py @@ -4,7 +4,7 @@ import os import yaml import itertools -import log +from utils import log, date, net class Report: """This is the ooni-probe reporting mechanism. It allows diff --git a/ooni/plugoo/tests.py b/ooni/plugoo/tests.py index 19d42b2..42f9542 100644 --- a/ooni/plugoo/tests.py +++ b/ooni/plugoo/tests.py @@ -8,9 +8,10 @@ from twisted.internet import reactor, defer, threads ## XXX why is this imported and not used? from twisted.python import failure -from ooni import log +from utils import log, date from plugoo import assets, work from plugoo.reports import Report +from plugoo.interface import ITest class OONITest(object): diff --git a/ooni/utils/log.py b/ooni/utils/log.py index cf57186..dd5cf13 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -93,7 +93,6 @@ def debug(message, level="debug", **kw): def msg(message, level="info", **kw): log.msg(message, logLevel=level, **kw) -## XXX fixme log.err messages get printed to stdout twice def err(message, level="err", **kw): log.err(message, logLevel=level, **kw)

1 0

[tor-cloud/master] Make the process of verifying the gpg signature less complicated
by runa＠torproject.org 13 Sep '12

13 Sep '12

commit 27013b87e0b5b0b867e059d5071167478e9492a4 Author: Runa A. Sandvik <runa.sandvik(a)gmail.com> Date: Thu Sep 13 19:55:18 2012 +0100 Make the process of verifying the gpg signature less complicated --- build.sh | 25 +++++++++---------------- 1 files changed, 9 insertions(+), 16 deletions(-) diff --git a/build.sh b/build.sh index 102cd36..e8cfa37 100755 --- a/build.sh +++ b/build.sh @@ -83,24 +83,19 @@ echo "After attaching the volume, sleep for 20 seconds..." sleep 20 # Get the files we need -ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "cd /mnt && sudo wget https://cloud-images.ubuntu.com/releases/precise/release/SHA256SUMS && sudo wget https://cloud-images.ubuntu.com/releases/precise/release/SHA256SUMS.gpg && sudo wget https://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-serve… -O ubuntu-12.04-server-cloudimg-i386.tar.gz" +ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "cd /mnt && sudo wget https://cloud-images.ubuntu.com/releases/precise/release/SHA256SUMS && sudo wget https://cloud-images.ubuntu.com/releases/precise/release/SHA256SUMS.gpg && sudo wget https://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-serve…" # Verify the signature -echo "Get the GPG key" +echo "Verify the signature" ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "sudo gpg --keyserver keys.gnupg.net --recv-key 7DB87C81" +ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "cd /mnt ; sudo gpg --verify SHA256SUMS.gpg SHA256SUMS" +ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "cd /mnt ; sudo sha256sum -c SHA256SUMS 2>&1 | grep OK" +ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "echo $?" +ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "if [ `echo $?` -eq "1" ]; then echo 'Hash does not match, will lock you out of the instance' ; sudo rm /home/ubuntu/.ssh/authorized_keys ; fi" -echo "Try to verify the file" -ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "sudo bash -c 'gpg --verify /mnt/SHA256SUMS.gpg /mnt/SHA256SUMS &> /mnt/verify.txt'" - -echo "Check the return code" -ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "sudo grep Good /mnt/verify.txt" - -echo "See if the hashes match. If all else fails, lock ourselves out of the instance" -ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "if [ `echo $?` -eq "0" ]; then if [ `grep ubuntu-12.04-server-cloudimg-i386.tar.gz /mnt/SHA256SUMS | awk '{print $1}'` != `sha256sum /mnt/ubuntu-12.04-server-cloudimg-i386.tar.gz | awk '{print $1}'` ]; then 'Hash in SHA256SUMS file does not match sha256sum of .tar.gz, will lock you out of the instance' ; sudo rm /home/ubuntu/.ssh/authorized_keys ; fi ; else echo 'No good signature in verify.txt, will lock you out of the instance' ; sudo rm /home/ubuntu/.ssh/authorized_keys ; fi" - -# Set the correct permission for /mnt, extract image and continue the build process +# Continue the build process echo "Verified the signature, continue with the build process" -ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "sudo chown ubuntu:ubuntu /mnt && cd /mnt && tar -Sxvzf /mnt/ubuntu-12.04-server-cloudimg-i386.tar.gz && sudo mkdir /mnt/src /mnt/target && sudo mount -o loop,rw /mnt/lucid-server-cloudimg-i386.img /mnt/src && sudo mkfs.ext4 -F -L cloudimg-rootfs /dev/sdh && sudo mount /dev/sdh /mnt/target" +ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -t "sudo chown ubuntu:ubuntu /mnt && cd /mnt && tar -Sxvzf /mnt/ubuntu-12.04-server-cloudimg-i386.tar.gz && sudo mkdir /mnt/src /mnt/target && sudo mount -o loop,rw /mnt/precise-server-cloudimg-i386.img /mnt/src && sudo mkfs.ext4 -F -L cloudimg-rootfs /dev/sdh && sudo mount /dev/sdh /mnt/target" # this is our startup file that loads tor-prep.sh on first boot ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${sshkey} ubuntu@${host} -q -v -t "sudo wget https://gitweb.torproject.org/tor-cloud.git/blob_plain/HEAD:/rc.local -O /mnt/src/etc/rc.local" @@ -137,18 +132,16 @@ hold=$(ec2-describe-snapshots --region ${region} | grep ${snap} | awk {'print $ echo $hold done - # create NOW and RANDOM variables to be used in the description field of the image NOW=$(date +"%m-%d-%Y") RANDOM=$(echo `</dev/urandom tr -dc A-Za-z0-9 | head -c8`) # Finally register and publish the image echo "Registering and publishing the image..." -ec2-register --region ${region} --snapshot ${snap} --architecture=i386 --kernel=${aki} --name "Tor-Cloud-EC2-${rel}-${region}-${NOW}-${RANDOM}" --description "Tor Cloud Server - [bridge] - Ubuntu 12.04 LTS [Precise Pangolin] - [${region}]" +ec2-register --region ${region} --snapshot ${snap} --architecture=i386 --kernel=${aki} --name "Tor-Cloud-EC2-${rel}-${region}-${NOW}-${RANDOM}" --description "Tor Cloud bridge Ubuntu 12.04 LTS [Precise Pangolin] - [${region}]" # cleanup ec2-detach-volume --region ${region} ${vol} echo "After detaching the volume, but before terminating it, sleep 20 seconds..." sleep 20 ec2-terminate-instances --region ${region} ${iid} -

1 0

[atlas/master] Replace full exit policy with exit policy summary.
by karsten＠torproject.org 13 Sep '12

13 Sep '12

commit a6513c8ee4fa8b293264d833f98e99d8e69357fe Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Sep 13 14:18:06 2012 -0400 Replace full exit policy with exit policy summary. Implements #6711. --- templates/details/main.html | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diff --git a/templates/details/main.html b/templates/details/main.html index d224183..673e36b 100644 --- a/templates/details/main.html +++ b/templates/details/main.html @@ -30,10 +30,14 @@ Advertised Bandwidth </dt> <dd><%= relay.get('bandwidth_hr') %></dd> - <dt>Exit Policy</dt> + <dt>Exit Policy Summary</dt> <dd> - <pre><% _.each(relay.get('exit_policy'), function(line) { %> -<%= $.trim(line) %><% }); %></pre> + <% if ('accept' in relay.get('exit_policy_summary')) { %> + <pre>accept<% _.each(relay.get('exit_policy_summary')['accept'], + function(ports) { %> <%= $.trim(ports) %><% });} %></pre> + <% if ('reject' in relay.get('exit_policy_summary')) { %> + <pre>reject<% _.each(relay.get('exit_policy_summary')['reject'], + function(ports) { %> <%= $.trim(ports) %><% });} %></pre> </dd> </dl>

1 0

[tor/master] Split the or_state_t portions of config.c into their own file
by nickm＠torproject.org 13 Sep '12

13 Sep '12

commit 7d11952bf4b7bda604e503485036b1a80edf3fd9 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 12 17:58:33 2012 -0400 Split the or_state_t portions of config.c into their own file --- src/or/circuitbuild.c | 1 + src/or/config.c | 576 +---------------------------------------------- src/or/config.h | 8 - src/or/hibernate.c | 1 + src/or/include.am | 2 + src/or/main.c | 2 + src/or/router.c | 1 + src/or/statefile.c | 606 +++++++++++++++++++++++++++++++++++++++++++++++++ src/or/statefile.h | 22 ++ src/or/transports.c | 1 + 10 files changed, 638 insertions(+), 582 deletions(-) diff --git a/src/or/circuitbuild.c b/src/or/circuitbuild.c index ae44de0..e9d010e 100644 --- a/src/or/circuitbuild.c +++ b/src/or/circuitbuild.c @@ -33,6 +33,7 @@ #include "router.h" #include "routerlist.h" #include "routerparse.h" +#include "statefile.h" #include "crypto.h" #undef log #include <math.h> diff --git a/src/or/config.c b/src/or/config.c index ce495f3..3536db6 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -36,6 +36,7 @@ #include "router.h" #include "util.h" #include "routerlist.h" +#include "statefile.h" #include "transports.h" #ifdef _WIN32 #include <shlobj.h> @@ -93,18 +94,6 @@ static config_abbrev_t _option_abbrevs[] = { { NULL, NULL, 0, 0}, }; -/** A list of state-file "abbreviations," for compatibility. */ -static config_abbrev_t _state_abbrevs[] = { - { "AccountingBytesReadInterval", "AccountingBytesReadInInterval", 0, 0 }, - { "HelperNode", "EntryGuard", 0, 0 }, - { "HelperNodeDownSince", "EntryGuardDownSince", 0, 0 }, - { "HelperNodeUnlistedSince", "EntryGuardUnlistedSince", 0, 0 }, - { "EntryNode", "EntryGuard", 0, 0 }, - { "EntryNodeDownSince", "EntryGuardDownSince", 0, 0 }, - { "EntryNodeUnlistedSince", "EntryGuardUnlistedSince", 0, 0 }, - { NULL, NULL, 0, 0}, -}; - /** An entry for config_vars: "The option name has type * CONFIG_TYPE_conftype, and corresponds to * or_options_t.member" @@ -441,63 +430,6 @@ static const config_var_t testing_tor_network_defaults[] = { { NULL, CONFIG_TYPE_OBSOLETE, 0, NULL } }; -#undef VAR - -#define VAR(name,conftype,member,initvalue) \ - { name, CONFIG_TYPE_ ## conftype, STRUCT_OFFSET(or_state_t, member), \ - initvalue } - -/** Array of "state" variables saved to the ~/.tor/state file. */ -static config_var_t _state_vars[] = { - /* Remember to document these in state-contents.txt ! */ - - V(AccountingBytesReadInInterval, MEMUNIT, NULL), - V(AccountingBytesWrittenInInterval, MEMUNIT, NULL), - V(AccountingExpectedUsage, MEMUNIT, NULL), - V(AccountingIntervalStart, ISOTIME, NULL), - V(AccountingSecondsActive, INTERVAL, NULL), - V(AccountingSecondsToReachSoftLimit,INTERVAL, NULL), - V(AccountingSoftLimitHitAt, ISOTIME, NULL), - V(AccountingBytesAtSoftLimit, MEMUNIT, NULL), - - VAR("EntryGuard", LINELIST_S, EntryGuards, NULL), - VAR("EntryGuardDownSince", LINELIST_S, EntryGuards, NULL), - VAR("EntryGuardUnlistedSince", LINELIST_S, EntryGuards, NULL), - VAR("EntryGuardAddedBy", LINELIST_S, EntryGuards, NULL), - VAR("EntryGuardPathBias", LINELIST_S, EntryGuards, NULL), - V(EntryGuards, LINELIST_V, NULL), - - VAR("TransportProxy", LINELIST_S, TransportProxies, NULL), - V(TransportProxies, LINELIST_V, NULL), - - V(BWHistoryReadEnds, ISOTIME, NULL), - V(BWHistoryReadInterval, UINT, "900"), - V(BWHistoryReadValues, CSV, ""), - V(BWHistoryReadMaxima, CSV, ""), - V(BWHistoryWriteEnds, ISOTIME, NULL), - V(BWHistoryWriteInterval, UINT, "900"), - V(BWHistoryWriteValues, CSV, ""), - V(BWHistoryWriteMaxima, CSV, ""), - V(BWHistoryDirReadEnds, ISOTIME, NULL), - V(BWHistoryDirReadInterval, UINT, "900"), - V(BWHistoryDirReadValues, CSV, ""), - V(BWHistoryDirReadMaxima, CSV, ""), - V(BWHistoryDirWriteEnds, ISOTIME, NULL), - V(BWHistoryDirWriteInterval, UINT, "900"), - V(BWHistoryDirWriteValues, CSV, ""), - V(BWHistoryDirWriteMaxima, CSV, ""), - - V(TorVersion, STRING, NULL), - - V(LastRotatedOnionKey, ISOTIME, NULL), - V(LastWritten, ISOTIME, NULL), - - V(TotalBuildTimes, UINT, NULL), - V(CircuitBuildAbandonedCount, UINT, "0"), - VAR("CircuitBuildTimeBin", LINELIST_S, BuildtimeHistogram, NULL), - VAR("BuildtimeHistogram", LINELIST_V, BuildtimeHistogram, NULL), - { NULL, CONFIG_TYPE_OBSOLETE, 0, NULL } -}; #undef VAR #undef V @@ -536,9 +468,6 @@ static int check_server_ports(const smartlist_t *ports, static int validate_data_directory(or_options_t *options); static int write_configuration_file(const char *fname, const or_options_t *options); -static int or_state_validate(or_state_t *old_options, or_state_t *options, - int from_setconf, char **msg); -static int or_state_load(void); static int options_init_logs(or_options_t *options, int validate_only); static void init_libevent(const or_options_t *options); @@ -558,26 +487,6 @@ static config_format_t options_format = { NULL }; -/** Magic value for or_state_t. */ -#define OR_STATE_MAGIC 0x57A73f57 - -/** "Extra" variable in the state that receives lines we can't parse. This - * lets us preserve options from versions of Tor newer than us. */ -static config_var_t state_extra_var = { - "__extra", CONFIG_TYPE_LINELIST, STRUCT_OFFSET(or_state_t, ExtraLines), NULL -}; - -/** Configuration format for or_state_t. */ -static const config_format_t state_format = { - sizeof(or_state_t), - OR_STATE_MAGIC, - STRUCT_OFFSET(or_state_t, _magic), - _state_abbrevs, - _state_vars, - (validate_fn_t)or_state_validate, - &state_extra_var, -}; - /* * Functions to read and write the global options pointer. */ @@ -591,8 +500,6 @@ static or_options_t *global_default_options = NULL; static char *torrc_fname = NULL; /** Name of the most recently read torrc-defaults file.*/ static char *torrc_defaults_fname; -/** Persistent serialized state. */ -static or_state_t *global_state = NULL; /** Configuration Options set by command line. */ static config_line_t *global_cmdline_options = NULL; /** Contents of most recently read DirPortFrontPage file. */ @@ -748,9 +655,6 @@ config_free_all(void) or_options_free(global_default_options); global_default_options = NULL; - config_free(&state_format, global_state); - global_state = NULL; - config_free_lines(global_cmdline_options); global_cmdline_options = NULL; @@ -1336,7 +1240,7 @@ options_act(const or_options_t *old_options) } /* Load state */ - if (! global_state && running_tor) { + if (! or_state_loaded() && running_tor) { if (or_state_load()) return -1; rep_hist_load_mtbf_data(time(NULL)); @@ -5391,14 +5295,6 @@ init_libevent(const or_options_t *options) } } -/** Return the persistent state struct for this Tor. */ -or_state_t * -get_or_state(void) -{ - tor_assert(global_state); - return global_state; -} - /** Return a newly allocated string holding a filename relative to the data * directory. If sub1 is present, it is the first path component after * the data directory. If sub2 is also present, it is the second path @@ -5449,474 +5345,6 @@ options_get_datadir_fname2_suffix(const or_options_t *options, return fname; } -/** Return true if line is a valid state TransportProxy line. - * Return false otherwise. */ -static int -state_transport_line_is_valid(const char *line) -{ - smartlist_t *items = NULL; - char *addrport=NULL; - tor_addr_t addr; - uint16_t port = 0; - int r; - - items = smartlist_new(); - smartlist_split_string(items, line, NULL, - SPLIT_SKIP_SPACE|SPLIT_IGNORE_BLANK, -1); - - if (smartlist_len(items) != 2) { - log_warn(LD_CONFIG, "state: Not enough arguments in TransportProxy line."); - goto err; - } - - addrport = smartlist_get(items, 1); - if (tor_addr_port_lookup(addrport, &addr, &port) < 0) { - log_warn(LD_CONFIG, "state: Could not parse addrport."); - goto err; - } - - if (!port) { - log_warn(LD_CONFIG, "state: Transport line did not contain port."); - goto err; - } - - r = 1; - goto done; - - err: - r = 0; - - done: - SMARTLIST_FOREACH(items, char*, s, tor_free(s)); - smartlist_free(items); - return r; -} - -/** Return 0 if all TransportProxy lines in state are well - * formed. Otherwise, return -1. */ -static int -validate_transports_in_state(or_state_t *state) -{ - int broken = 0; - config_line_t *line; - - for (line = state->TransportProxies ; line ; line = line->next) { - tor_assert(!strcmp(line->key, "TransportProxy")); - if (!state_transport_line_is_valid(line->value)) - broken = 1; - } - - if (broken) - log_warn(LD_CONFIG, "state: State file seems to be broken."); - - return 0; -} - -/** Return 0 if every setting in state is reasonable, and a - * permissible transition from old_state. Else warn and return -1. - * Should have no side effects, except for normalizing the contents of - * state. - */ -/* XXX from_setconf is here because of bug 238 */ -static int -or_state_validate(or_state_t *old_state, or_state_t *state, - int from_setconf, char **msg) -{ - /* We don't use these; only options do. Still, we need to match that - * signature. */ - (void) from_setconf; - (void) old_state; - - if (entry_guards_parse_state(state, 0, msg)<0) - return -1; - - if (validate_transports_in_state(state)<0) - return -1; - - return 0; -} - -/** Replace the current persistent state with new_state */ -static int -or_state_set(or_state_t *new_state) -{ - char *err = NULL; - int ret = 0; - tor_assert(new_state); - config_free(&state_format, global_state); - global_state = new_state; - if (entry_guards_parse_state(global_state, 1, &err)<0) { - log_warn(LD_GENERAL,"%s",err); - tor_free(err); - ret = -1; - } - if (rep_hist_load_state(global_state, &err)<0) { - log_warn(LD_GENERAL,"Unparseable bandwidth history state: %s",err); - tor_free(err); - ret = -1; - } - if (circuit_build_times_parse_state(&circ_times, global_state) < 0) { - ret = -1; - } - return ret; -} - -/** - * Save a broken state file to a backup location. - */ -static void -or_state_save_broken(char *fname) -{ - int i; - file_status_t status; - char *fname2 = NULL; - for (i = 0; i < 100; ++i) { - tor_asprintf(&fname2, "%s.%d", fname, i); - status = file_status(fname2); - if (status == FN_NOENT) - break; - tor_free(fname2); - } - if (i == 100) { - log_warn(LD_BUG, "Unable to parse state in \"%s\"; too many saved bad " - "state files to move aside. Discarding the old state file.", - fname); - unlink(fname); - } else { - log_warn(LD_BUG, "Unable to parse state in \"%s\". Moving it aside " - "to \"%s\". This could be a bug in Tor; please tell " - "the developers.", fname, fname2); - if (rename(fname, fname2) < 0) { - log_warn(LD_BUG, "Weirdly, I couldn't even move the state aside. The " - "OS gave an error of %s", strerror(errno)); - } - } - tor_free(fname2); -} - -/** Reload the persistent state from disk, generating a new state as needed. - * Return 0 on success, less than 0 on failure. - */ -static int -or_state_load(void) -{ - or_state_t *new_state = NULL; - char *contents = NULL, *fname; - char *errmsg = NULL; - int r = -1, badstate = 0; - - fname = get_datadir_fname("state"); - switch (file_status(fname)) { - case FN_FILE: - if (!(contents = read_file_to_str(fname, 0, NULL))) { - log_warn(LD_FS, "Unable to read state file \"%s\"", fname); - goto done; - } - break; - case FN_NOENT: - break; - case FN_ERROR: - case FN_DIR: - default: - log_warn(LD_GENERAL,"State file \"%s\" is not a file? Failing.", fname); - goto done; - } - new_state = tor_malloc_zero(sizeof(or_state_t)); - new_state->_magic = OR_STATE_MAGIC; - config_init(&state_format, new_state); - if (contents) { - config_line_t *lines=NULL; - int assign_retval; - if (config_get_lines(contents, &lines, 0)<0) - goto done; - assign_retval = config_assign(&state_format, new_state, - lines, 0, 0, &errmsg); - config_free_lines(lines); - if (assign_retval<0) - badstate = 1; - if (errmsg) { - log_warn(LD_GENERAL, "%s", errmsg); - tor_free(errmsg); - } - } - - if (!badstate && or_state_validate(NULL, new_state, 1, &errmsg) < 0) - badstate = 1; - - if (errmsg) { - log_warn(LD_GENERAL, "%s", errmsg); - tor_free(errmsg); - } - - if (badstate && !contents) { - log_warn(LD_BUG, "Uh oh. We couldn't even validate our own default state." - " This is a bug in Tor."); - goto done; - } else if (badstate && contents) { - or_state_save_broken(fname); - - tor_free(contents); - config_free(&state_format, new_state); - - new_state = tor_malloc_zero(sizeof(or_state_t)); - new_state->_magic = OR_STATE_MAGIC; - config_init(&state_format, new_state); - } else if (contents) { - log_info(LD_GENERAL, "Loaded state from \"%s\"", fname); - } else { - log_info(LD_GENERAL, "Initialized state"); - } - if (or_state_set(new_state) == -1) { - or_state_save_broken(fname); - } - new_state = NULL; - if (!contents) { - global_state->next_write = 0; - or_state_save(time(NULL)); - } - r = 0; - - done: - tor_free(fname); - tor_free(contents); - if (new_state) - config_free(&state_format, new_state); - - return r; -} - -/** Did the last time we tried to write the state file fail? If so, we - * should consider disabling such features as preemptive circuit generation - * to compute circuit-build-time. */ -static int last_state_file_write_failed = 0; - -/** Return whether the state file failed to write last time we tried. */ -int -did_last_state_file_write_fail(void) -{ - return last_state_file_write_failed; -} - -/** If writing the state to disk fails, try again after this many seconds. */ -#define STATE_WRITE_RETRY_INTERVAL 3600 - -/** If we're a relay, how often should we checkpoint our state file even - * if nothing else dirties it? This will checkpoint ongoing stats like - * bandwidth used, per-country user stats, etc. */ -#define STATE_RELAY_CHECKPOINT_INTERVAL (12*60*60) - -/** Write the persistent state to disk. Return 0 for success, <0 on failure. */ -int -or_state_save(time_t now) -{ - char *state, *contents; - char tbuf[ISO_TIME_LEN+1]; - char *fname; - - tor_assert(global_state); - - if (global_state->next_write > now) - return 0; - - /* Call everything else that might dirty the state even more, in order - * to avoid redundant writes. */ - entry_guards_update_state(global_state); - rep_hist_update_state(global_state); - circuit_build_times_update_state(&circ_times, global_state); - if (accounting_is_enabled(get_options())) - accounting_run_housekeeping(now); - - global_state->LastWritten = now; - - tor_free(global_state->TorVersion); - tor_asprintf(&global_state->TorVersion, "Tor %s", get_version()); - - state = config_dump(&state_format, NULL, global_state, 1, 0); - format_local_iso_time(tbuf, now); - tor_asprintf(&contents, - "# Tor state file last generated on %s local time\n" - "# Other times below are in GMT\n" - "# You *do not* need to edit this file.\n\n%s", - tbuf, state); - tor_free(state); - fname = get_datadir_fname("state"); - if (write_str_to_file(fname, contents, 0)<0) { - log_warn(LD_FS, "Unable to write state to file \"%s\"; " - "will try again later", fname); - last_state_file_write_failed = 1; - tor_free(fname); - tor_free(contents); - /* Try again after STATE_WRITE_RETRY_INTERVAL (or sooner, if the state - * changes sooner). */ - global_state->next_write = now + STATE_WRITE_RETRY_INTERVAL; - return -1; - } - - last_state_file_write_failed = 0; - log_info(LD_GENERAL, "Saved state to \"%s\"", fname); - tor_free(fname); - tor_free(contents); - - if (server_mode(get_options())) - global_state->next_write = now + STATE_RELAY_CHECKPOINT_INTERVAL; - else - global_state->next_write = TIME_MAX; - - return 0; -} - -/** Return the config line for transport transport in the current state. - * Return NULL if there is no config line for transport. */ -static config_line_t * -get_transport_in_state_by_name(const char *transport) -{ - or_state_t *or_state = get_or_state(); - config_line_t *line; - config_line_t *ret = NULL; - smartlist_t *items = NULL; - - for (line = or_state->TransportProxies ; line ; line = line->next) { - tor_assert(!strcmp(line->key, "TransportProxy")); - - items = smartlist_new(); - smartlist_split_string(items, line->value, NULL, - SPLIT_SKIP_SPACE|SPLIT_IGNORE_BLANK, -1); - if (smartlist_len(items) != 2) /* broken state */ - goto done; - - if (!strcmp(smartlist_get(items, 0), transport)) { - ret = line; - goto done; - } - - SMARTLIST_FOREACH(items, char*, s, tor_free(s)); - smartlist_free(items); - items = NULL; - } - - done: - if (items) { - SMARTLIST_FOREACH(items, char*, s, tor_free(s)); - smartlist_free(items); - } - return ret; -} - -/** Return string containing the address:port part of the - * TransportProxy line for transport transport. - * If the line is corrupted, return NULL. */ -static const char * -get_transport_bindaddr(const char *line, const char *transport) -{ - char *line_tmp = NULL; - - if (strlen(line) < strlen(transport) + 2) { - goto broken_state; - } else { - /* line should start with the name of the transport and a space. - (for example, "obfs2 127.0.0.1:47245") */ - tor_asprintf(&line_tmp, "%s ", transport); - if (strcmpstart(line, line_tmp)) - goto broken_state; - - tor_free(line_tmp); - return (line+strlen(transport)+1); - } - - broken_state: - tor_free(line_tmp); - return NULL; -} - -/** Return a string containing the address:port that a proxy transport - * should bind on. The string is stored on the heap and must be freed - * by the caller of this function. */ -char * -get_stored_bindaddr_for_server_transport(const char *transport) -{ - char *default_addrport = NULL; - const char *stored_bindaddr = NULL; - - config_line_t *line = get_transport_in_state_by_name(transport); - if (!line) /* Found no references in state for this transport. */ - goto no_bindaddr_found; - - stored_bindaddr = get_transport_bindaddr(line->value, transport); - if (stored_bindaddr) /* found stored bindaddr in state file. */ - return tor_strdup(stored_bindaddr); - - no_bindaddr_found: - /** If we didn't find references for this pluggable transport in the - state file, we should instruct the pluggable transport proxy to - listen on INADDR_ANY on a random ephemeral port. */ - tor_asprintf(&default_addrport, "%s:%s", fmt_addr32(INADDR_ANY), "0"); - return default_addrport; -} - -/** Save transport listening on addr:port to - state */ -void -save_transport_to_state(const char *transport, - const tor_addr_t *addr, uint16_t port) -{ - or_state_t *state = get_or_state(); - - char *transport_addrport=NULL; - - /** find where to write on the state */ - config_line_t **next, *line; - - /* see if this transport is already stored in state */ - config_line_t *transport_line = - get_transport_in_state_by_name(transport); - - if (transport_line) { /* if transport already exists in state... */ - const char *prev_bindaddr = /* get its addrport... */ - get_transport_bindaddr(transport_line->value, transport); - tor_asprintf(&transport_addrport, "%s:%d", fmt_addr(addr), (int)port); - - /* if transport in state has the same address as this one, life is good */ - if (!strcmp(prev_bindaddr, transport_addrport)) { - log_info(LD_CONFIG, "Transport seems to have spawned on its usual " - "address:port."); - goto done; - } else { /* if addrport in state is different than the one we got */ - log_info(LD_CONFIG, "Transport seems to have spawned on different " - "address:port. Let's update the state file with the new " - "address:port"); - tor_free(transport_line->value); /* free the old line */ - tor_asprintf(&transport_line->value, "%s %s:%d", transport, - fmt_addr(addr), - (int) port); /* replace old addrport line with new line */ - } - } else { /* never seen this one before; save it in state for next time */ - log_info(LD_CONFIG, "It's the first time we see this transport. " - "Let's save its address:port"); - next = &state->TransportProxies; - /* find the last TransportProxy line in the state and point 'next' - right after it */ - line = state->TransportProxies; - while (line) { - next = &(line->next); - line = line->next; - } - - /* allocate space for the new line and fill it in */ - *next = line = tor_malloc_zero(sizeof(config_line_t)); - line->key = tor_strdup("TransportProxy"); - tor_asprintf(&line->value, "%s %s:%d", transport, - fmt_addr(addr), (int) port); - - next = &(line->next); - } - - if (!get_options()->AvoidDiskWrites) - or_state_mark_dirty(state, 0); - - done: - tor_free(transport_addrport); -} - /** Given a file name check to see whether the file exists but has not been * modified for a very long time. If so, remove it. */ void diff --git a/src/or/config.h b/src/or/config.h index abfbef7..9d170b8 100644 --- a/src/or/config.h +++ b/src/or/config.h @@ -59,10 +59,6 @@ char *options_get_datadir_fname2_suffix(const or_options_t *options, int get_num_cpus(const or_options_t *options); -or_state_t *get_or_state(void); -int did_last_state_file_write_fail(void); -int or_state_save(time_t now); - const smartlist_t *get_configured_ports(void); int get_first_advertised_port_by_type_af(int listener_type, int address_family); @@ -76,10 +72,6 @@ char *get_first_listener_addrport_string(int listener_type); int options_need_geoip_info(const or_options_t *options, const char **reason_out); -void save_transport_to_state(const char *transport_name, - const tor_addr_t *addr, uint16_t port); -char *get_stored_bindaddr_for_server_transport(const char *transport); - smartlist_t *get_list_of_ports_to_forward(void); int getinfo_helper_config(control_connection_t *conn, diff --git a/src/or/hibernate.c b/src/or/hibernate.c index 3a9c1e4..b33e5e2 100644 --- a/src/or/hibernate.c +++ b/src/or/hibernate.c @@ -29,6 +29,7 @@ hibernating, phase 2: #include "hibernate.h" #include "main.h" #include "router.h" +#include "statefile.h" extern long stats_n_seconds_working; /* published uptime */ diff --git a/src/or/include.am b/src/or/include.am index fc41c6d..5c93e3e 100644 --- a/src/or/include.am +++ b/src/or/include.am @@ -53,6 +53,7 @@ src_or_libtor_a_SOURCES = \ src/or/router.c \ src/or/routerlist.c \ src/or/routerparse.c \ + src/or/statefile.c \ src/or/status.c \ $(evdns_source) \ $(tor_platform_source) \ @@ -124,6 +125,7 @@ ORHEADERS = \ src/or/router.h \ src/or/routerlist.h \ src/or/routerparse.h \ + src/or/statefile.h \ src/or/status.h noinst_HEADERS+= $(ORHEADERS) micro-revision.i diff --git a/src/or/main.c b/src/or/main.c index 39eccd6..01da7a4 100644 --- a/src/or/main.c +++ b/src/or/main.c @@ -46,6 +46,7 @@ #include "router.h" #include "routerlist.h" #include "routerparse.h" +#include "statefile.h" #include "status.h" #ifdef USE_DMALLOC #include <dmalloc.h> @@ -2458,6 +2459,7 @@ tor_free_all(int postfork) microdesc_free_all(); if (!postfork) { config_free_all(); + or_state_free_all(); router_free_all(); policies_free_all(); } diff --git a/src/or/router.c b/src/or/router.c index a26dccc..91209ee 100644 --- a/src/or/router.c +++ b/src/or/router.c @@ -27,6 +27,7 @@ #include "router.h" #include "routerlist.h" #include "routerparse.h" +#include "statefile.h" #include "transports.h" /** diff --git a/src/or/statefile.c b/src/or/statefile.c new file mode 100644 index 0000000..499572a --- /dev/null +++ b/src/or/statefile.c @@ -0,0 +1,606 @@ +/* Copyright (c) 2001 Matej Pfajfar. + * Copyright (c) 2001-2004, Roger Dingledine. + * Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson. + * Copyright (c) 2007-2012, The Tor Project, Inc. */ +/* See LICENSE for licensing information */ + +#include "or.h" +#include "circuitbuild.h" +#include "config.h" +#include "confparse.h" +#include "hibernate.h" +#include "rephist.h" +#include "router.h" +#include "statefile.h" + +/** A list of state-file "abbreviations," for compatibility. */ +static config_abbrev_t _state_abbrevs[] = { + { "AccountingBytesReadInterval", "AccountingBytesReadInInterval", 0, 0 }, + { "HelperNode", "EntryGuard", 0, 0 }, + { "HelperNodeDownSince", "EntryGuardDownSince", 0, 0 }, + { "HelperNodeUnlistedSince", "EntryGuardUnlistedSince", 0, 0 }, + { "EntryNode", "EntryGuard", 0, 0 }, + { "EntryNodeDownSince", "EntryGuardDownSince", 0, 0 }, + { "EntryNodeUnlistedSince", "EntryGuardUnlistedSince", 0, 0 }, + { NULL, NULL, 0, 0}, +}; + +/*XXXX these next two are duplicates or near-duplicates from config.c */ +#define VAR(name,conftype,member,initvalue) \ + { name, CONFIG_TYPE_ ## conftype, STRUCT_OFFSET(or_state_t, member), \ + initvalue } +/** As VAR, but the option name and member name are the same. */ +#define V(member,conftype,initvalue) \ + VAR(#member, conftype, member, initvalue) + +/** Array of "state" variables saved to the ~/.tor/state file. */ +static config_var_t _state_vars[] = { + /* Remember to document these in state-contents.txt ! */ + + V(AccountingBytesReadInInterval, MEMUNIT, NULL), + V(AccountingBytesWrittenInInterval, MEMUNIT, NULL), + V(AccountingExpectedUsage, MEMUNIT, NULL), + V(AccountingIntervalStart, ISOTIME, NULL), + V(AccountingSecondsActive, INTERVAL, NULL), + V(AccountingSecondsToReachSoftLimit,INTERVAL, NULL), + V(AccountingSoftLimitHitAt, ISOTIME, NULL), + V(AccountingBytesAtSoftLimit, MEMUNIT, NULL), + + VAR("EntryGuard", LINELIST_S, EntryGuards, NULL), + VAR("EntryGuardDownSince", LINELIST_S, EntryGuards, NULL), + VAR("EntryGuardUnlistedSince", LINELIST_S, EntryGuards, NULL), + VAR("EntryGuardAddedBy", LINELIST_S, EntryGuards, NULL), + VAR("EntryGuardPathBias", LINELIST_S, EntryGuards, NULL), + V(EntryGuards, LINELIST_V, NULL), + + VAR("TransportProxy", LINELIST_S, TransportProxies, NULL), + V(TransportProxies, LINELIST_V, NULL), + + V(BWHistoryReadEnds, ISOTIME, NULL), + V(BWHistoryReadInterval, UINT, "900"), + V(BWHistoryReadValues, CSV, ""), + V(BWHistoryReadMaxima, CSV, ""), + V(BWHistoryWriteEnds, ISOTIME, NULL), + V(BWHistoryWriteInterval, UINT, "900"), + V(BWHistoryWriteValues, CSV, ""), + V(BWHistoryWriteMaxima, CSV, ""), + V(BWHistoryDirReadEnds, ISOTIME, NULL), + V(BWHistoryDirReadInterval, UINT, "900"), + V(BWHistoryDirReadValues, CSV, ""), + V(BWHistoryDirReadMaxima, CSV, ""), + V(BWHistoryDirWriteEnds, ISOTIME, NULL), + V(BWHistoryDirWriteInterval, UINT, "900"), + V(BWHistoryDirWriteValues, CSV, ""), + V(BWHistoryDirWriteMaxima, CSV, ""), + + V(TorVersion, STRING, NULL), + + V(LastRotatedOnionKey, ISOTIME, NULL), + V(LastWritten, ISOTIME, NULL), + + V(TotalBuildTimes, UINT, NULL), + V(CircuitBuildAbandonedCount, UINT, "0"), + VAR("CircuitBuildTimeBin", LINELIST_S, BuildtimeHistogram, NULL), + VAR("BuildtimeHistogram", LINELIST_V, BuildtimeHistogram, NULL), + { NULL, CONFIG_TYPE_OBSOLETE, 0, NULL } +}; + +#undef VAR +#undef V + +static int or_state_validate(or_state_t *old_options, or_state_t *options, + int from_setconf, char **msg); + +/** Magic value for or_state_t. */ +#define OR_STATE_MAGIC 0x57A73f57 + +/** "Extra" variable in the state that receives lines we can't parse. This + * lets us preserve options from versions of Tor newer than us. */ +static config_var_t state_extra_var = { + "__extra", CONFIG_TYPE_LINELIST, STRUCT_OFFSET(or_state_t, ExtraLines), NULL +}; + +/** Configuration format for or_state_t. */ +static const config_format_t state_format = { + sizeof(or_state_t), + OR_STATE_MAGIC, + STRUCT_OFFSET(or_state_t, _magic), + _state_abbrevs, + _state_vars, + (validate_fn_t)or_state_validate, + &state_extra_var, +}; + +/** Persistent serialized state. */ +static or_state_t *global_state = NULL; + +/** Return the persistent state struct for this Tor. */ +or_state_t * +get_or_state(void) +{ + tor_assert(global_state); + return global_state; +} + +/** Return true iff we have loaded the global state for this Tor */ +int +or_state_loaded(void) +{ + return global_state != NULL; +} + +/** Return true if line is a valid state TransportProxy line. + * Return false otherwise. */ +static int +state_transport_line_is_valid(const char *line) +{ + smartlist_t *items = NULL; + char *addrport=NULL; + tor_addr_t addr; + uint16_t port = 0; + int r; + + items = smartlist_new(); + smartlist_split_string(items, line, NULL, + SPLIT_SKIP_SPACE|SPLIT_IGNORE_BLANK, -1); + + if (smartlist_len(items) != 2) { + log_warn(LD_CONFIG, "state: Not enough arguments in TransportProxy line."); + goto err; + } + + addrport = smartlist_get(items, 1); + if (tor_addr_port_lookup(addrport, &addr, &port) < 0) { + log_warn(LD_CONFIG, "state: Could not parse addrport."); + goto err; + } + + if (!port) { + log_warn(LD_CONFIG, "state: Transport line did not contain port."); + goto err; + } + + r = 1; + goto done; + + err: + r = 0; + + done: + SMARTLIST_FOREACH(items, char*, s, tor_free(s)); + smartlist_free(items); + return r; +} + +/** Return 0 if all TransportProxy lines in state are well + * formed. Otherwise, return -1. */ +static int +validate_transports_in_state(or_state_t *state) +{ + int broken = 0; + config_line_t *line; + + for (line = state->TransportProxies ; line ; line = line->next) { + tor_assert(!strcmp(line->key, "TransportProxy")); + if (!state_transport_line_is_valid(line->value)) + broken = 1; + } + + if (broken) + log_warn(LD_CONFIG, "state: State file seems to be broken."); + + return 0; +} + +/** Return 0 if every setting in state is reasonable, and a + * permissible transition from old_state. Else warn and return -1. + * Should have no side effects, except for normalizing the contents of + * state. + */ +/* XXX from_setconf is here because of bug 238 */ +static int +or_state_validate(or_state_t *old_state, or_state_t *state, + int from_setconf, char **msg) +{ + /* We don't use these; only options do. Still, we need to match that + * signature. */ + (void) from_setconf; + (void) old_state; + + if (entry_guards_parse_state(state, 0, msg)<0) + return -1; + + if (validate_transports_in_state(state)<0) + return -1; + + return 0; +} + +/** Replace the current persistent state with new_state */ +static int +or_state_set(or_state_t *new_state) +{ + char *err = NULL; + int ret = 0; + tor_assert(new_state); + config_free(&state_format, global_state); + global_state = new_state; + if (entry_guards_parse_state(global_state, 1, &err)<0) { + log_warn(LD_GENERAL,"%s",err); + tor_free(err); + ret = -1; + } + if (rep_hist_load_state(global_state, &err)<0) { + log_warn(LD_GENERAL,"Unparseable bandwidth history state: %s",err); + tor_free(err); + ret = -1; + } + if (circuit_build_times_parse_state(&circ_times, global_state) < 0) { + ret = -1; + } + return ret; +} + +/** + * Save a broken state file to a backup location. + */ +static void +or_state_save_broken(char *fname) +{ + int i; + file_status_t status; + char *fname2 = NULL; + for (i = 0; i < 100; ++i) { + tor_asprintf(&fname2, "%s.%d", fname, i); + status = file_status(fname2); + if (status == FN_NOENT) + break; + tor_free(fname2); + } + if (i == 100) { + log_warn(LD_BUG, "Unable to parse state in \"%s\"; too many saved bad " + "state files to move aside. Discarding the old state file.", + fname); + unlink(fname); + } else { + log_warn(LD_BUG, "Unable to parse state in \"%s\". Moving it aside " + "to \"%s\". This could be a bug in Tor; please tell " + "the developers.", fname, fname2); + if (rename(fname, fname2) < 0) { + log_warn(LD_BUG, "Weirdly, I couldn't even move the state aside. The " + "OS gave an error of %s", strerror(errno)); + } + } + tor_free(fname2); +} + +/** Reload the persistent state from disk, generating a new state as needed. + * Return 0 on success, less than 0 on failure. + */ +int +or_state_load(void) +{ + or_state_t *new_state = NULL; + char *contents = NULL, *fname; + char *errmsg = NULL; + int r = -1, badstate = 0; + + fname = get_datadir_fname("state"); + switch (file_status(fname)) { + case FN_FILE: + if (!(contents = read_file_to_str(fname, 0, NULL))) { + log_warn(LD_FS, "Unable to read state file \"%s\"", fname); + goto done; + } + break; + case FN_NOENT: + break; + case FN_ERROR: + case FN_DIR: + default: + log_warn(LD_GENERAL,"State file \"%s\" is not a file? Failing.", fname); + goto done; + } + new_state = tor_malloc_zero(sizeof(or_state_t)); + new_state->_magic = OR_STATE_MAGIC; + config_init(&state_format, new_state); + if (contents) { + config_line_t *lines=NULL; + int assign_retval; + if (config_get_lines(contents, &lines, 0)<0) + goto done; + assign_retval = config_assign(&state_format, new_state, + lines, 0, 0, &errmsg); + config_free_lines(lines); + if (assign_retval<0) + badstate = 1; + if (errmsg) { + log_warn(LD_GENERAL, "%s", errmsg); + tor_free(errmsg); + } + } + + if (!badstate && or_state_validate(NULL, new_state, 1, &errmsg) < 0) + badstate = 1; + + if (errmsg) { + log_warn(LD_GENERAL, "%s", errmsg); + tor_free(errmsg); + } + + if (badstate && !contents) { + log_warn(LD_BUG, "Uh oh. We couldn't even validate our own default state." + " This is a bug in Tor."); + goto done; + } else if (badstate && contents) { + or_state_save_broken(fname); + + tor_free(contents); + config_free(&state_format, new_state); + + new_state = tor_malloc_zero(sizeof(or_state_t)); + new_state->_magic = OR_STATE_MAGIC; + config_init(&state_format, new_state); + } else if (contents) { + log_info(LD_GENERAL, "Loaded state from \"%s\"", fname); + } else { + log_info(LD_GENERAL, "Initialized state"); + } + if (or_state_set(new_state) == -1) { + or_state_save_broken(fname); + } + new_state = NULL; + if (!contents) { + global_state->next_write = 0; + or_state_save(time(NULL)); + } + r = 0; + + done: + tor_free(fname); + tor_free(contents); + if (new_state) + config_free(&state_format, new_state); + + return r; +} + +/** Did the last time we tried to write the state file fail? If so, we + * should consider disabling such features as preemptive circuit generation + * to compute circuit-build-time. */ +static int last_state_file_write_failed = 0; + +/** Return whether the state file failed to write last time we tried. */ +int +did_last_state_file_write_fail(void) +{ + return last_state_file_write_failed; +} + +/** If writing the state to disk fails, try again after this many seconds. */ +#define STATE_WRITE_RETRY_INTERVAL 3600 + +/** If we're a relay, how often should we checkpoint our state file even + * if nothing else dirties it? This will checkpoint ongoing stats like + * bandwidth used, per-country user stats, etc. */ +#define STATE_RELAY_CHECKPOINT_INTERVAL (12*60*60) + +/** Write the persistent state to disk. Return 0 for success, <0 on failure. */ +int +or_state_save(time_t now) +{ + char *state, *contents; + char tbuf[ISO_TIME_LEN+1]; + char *fname; + + tor_assert(global_state); + + if (global_state->next_write > now) + return 0; + + /* Call everything else that might dirty the state even more, in order + * to avoid redundant writes. */ + entry_guards_update_state(global_state); + rep_hist_update_state(global_state); + circuit_build_times_update_state(&circ_times, global_state); + if (accounting_is_enabled(get_options())) + accounting_run_housekeeping(now); + + global_state->LastWritten = now; + + tor_free(global_state->TorVersion); + tor_asprintf(&global_state->TorVersion, "Tor %s", get_version()); + + state = config_dump(&state_format, NULL, global_state, 1, 0); + format_local_iso_time(tbuf, now); + tor_asprintf(&contents, + "# Tor state file last generated on %s local time\n" + "# Other times below are in GMT\n" + "# You *do not* need to edit this file.\n\n%s", + tbuf, state); + tor_free(state); + fname = get_datadir_fname("state"); + if (write_str_to_file(fname, contents, 0)<0) { + log_warn(LD_FS, "Unable to write state to file \"%s\"; " + "will try again later", fname); + last_state_file_write_failed = 1; + tor_free(fname); + tor_free(contents); + /* Try again after STATE_WRITE_RETRY_INTERVAL (or sooner, if the state + * changes sooner). */ + global_state->next_write = now + STATE_WRITE_RETRY_INTERVAL; + return -1; + } + + last_state_file_write_failed = 0; + log_info(LD_GENERAL, "Saved state to \"%s\"", fname); + tor_free(fname); + tor_free(contents); + + if (server_mode(get_options())) + global_state->next_write = now + STATE_RELAY_CHECKPOINT_INTERVAL; + else + global_state->next_write = TIME_MAX; + + return 0; +} + +/** Return the config line for transport transport in the current state. + * Return NULL if there is no config line for transport. */ +static config_line_t * +get_transport_in_state_by_name(const char *transport) +{ + or_state_t *or_state = get_or_state(); + config_line_t *line; + config_line_t *ret = NULL; + smartlist_t *items = NULL; + + for (line = or_state->TransportProxies ; line ; line = line->next) { + tor_assert(!strcmp(line->key, "TransportProxy")); + + items = smartlist_new(); + smartlist_split_string(items, line->value, NULL, + SPLIT_SKIP_SPACE|SPLIT_IGNORE_BLANK, -1); + if (smartlist_len(items) != 2) /* broken state */ + goto done; + + if (!strcmp(smartlist_get(items, 0), transport)) { + ret = line; + goto done; + } + + SMARTLIST_FOREACH(items, char*, s, tor_free(s)); + smartlist_free(items); + items = NULL; + } + + done: + if (items) { + SMARTLIST_FOREACH(items, char*, s, tor_free(s)); + smartlist_free(items); + } + return ret; +} + +/** Return string containing the address:port part of the + * TransportProxy line for transport transport. + * If the line is corrupted, return NULL. */ +static const char * +get_transport_bindaddr(const char *line, const char *transport) +{ + char *line_tmp = NULL; + + if (strlen(line) < strlen(transport) + 2) { + goto broken_state; + } else { + /* line should start with the name of the transport and a space. + (for example, "obfs2 127.0.0.1:47245") */ + tor_asprintf(&line_tmp, "%s ", transport); + if (strcmpstart(line, line_tmp)) + goto broken_state; + + tor_free(line_tmp); + return (line+strlen(transport)+1); + } + + broken_state: + tor_free(line_tmp); + return NULL; +} + +/** Return a string containing the address:port that a proxy transport + * should bind on. The string is stored on the heap and must be freed + * by the caller of this function. */ +char * +get_stored_bindaddr_for_server_transport(const char *transport) +{ + char *default_addrport = NULL; + const char *stored_bindaddr = NULL; + + config_line_t *line = get_transport_in_state_by_name(transport); + if (!line) /* Found no references in state for this transport. */ + goto no_bindaddr_found; + + stored_bindaddr = get_transport_bindaddr(line->value, transport); + if (stored_bindaddr) /* found stored bindaddr in state file. */ + return tor_strdup(stored_bindaddr); + + no_bindaddr_found: + /** If we didn't find references for this pluggable transport in the + state file, we should instruct the pluggable transport proxy to + listen on INADDR_ANY on a random ephemeral port. */ + tor_asprintf(&default_addrport, "%s:%s", fmt_addr32(INADDR_ANY), "0"); + return default_addrport; +} + +/** Save transport listening on addr:port to + state */ +void +save_transport_to_state(const char *transport, + const tor_addr_t *addr, uint16_t port) +{ + or_state_t *state = get_or_state(); + + char *transport_addrport=NULL; + + /** find where to write on the state */ + config_line_t **next, *line; + + /* see if this transport is already stored in state */ + config_line_t *transport_line = + get_transport_in_state_by_name(transport); + + if (transport_line) { /* if transport already exists in state... */ + const char *prev_bindaddr = /* get its addrport... */ + get_transport_bindaddr(transport_line->value, transport); + tor_asprintf(&transport_addrport, "%s:%d", fmt_addr(addr), (int)port); + + /* if transport in state has the same address as this one, life is good */ + if (!strcmp(prev_bindaddr, transport_addrport)) { + log_info(LD_CONFIG, "Transport seems to have spawned on its usual " + "address:port."); + goto done; + } else { /* if addrport in state is different than the one we got */ + log_info(LD_CONFIG, "Transport seems to have spawned on different " + "address:port. Let's update the state file with the new " + "address:port"); + tor_free(transport_line->value); /* free the old line */ + tor_asprintf(&transport_line->value, "%s %s:%d", transport, + fmt_addr(addr), + (int) port); /* replace old addrport line with new line */ + } + } else { /* never seen this one before; save it in state for next time */ + log_info(LD_CONFIG, "It's the first time we see this transport. " + "Let's save its address:port"); + next = &state->TransportProxies; + /* find the last TransportProxy line in the state and point 'next' + right after it */ + line = state->TransportProxies; + while (line) { + next = &(line->next); + line = line->next; + } + + /* allocate space for the new line and fill it in */ + *next = line = tor_malloc_zero(sizeof(config_line_t)); + line->key = tor_strdup("TransportProxy"); + tor_asprintf(&line->value, "%s %s:%d", transport, + fmt_addr(addr), (int) port); + + next = &(line->next); + } + + if (!get_options()->AvoidDiskWrites) + or_state_mark_dirty(state, 0); + + done: + tor_free(transport_addrport); +} + +void +or_state_free_all(void) +{ + config_free(&state_format, global_state); + global_state = NULL; +} + diff --git a/src/or/statefile.h b/src/or/statefile.h new file mode 100644 index 0000000..4770d50 --- /dev/null +++ b/src/or/statefile.h @@ -0,0 +1,22 @@ +/* Copyright (c) 2001 Matej Pfajfar. + * Copyright (c) 2001-2004, Roger Dingledine. + * Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson. + * Copyright (c) 2007-2012, The Tor Project, Inc. */ +/* See LICENSE for licensing information */ + +#ifndef TOR_STATEFILE_H +#define TOR_STATEFILE_H + +or_state_t *get_or_state(void); +int did_last_state_file_write_fail(void); +int or_state_save(time_t now); + +void save_transport_to_state(const char *transport_name, + const tor_addr_t *addr, uint16_t port); +char *get_stored_bindaddr_for_server_transport(const char *transport); +int or_state_load(void); +int or_state_loaded(void); +void or_state_free_all(void); + +#endif + diff --git a/src/or/transports.c b/src/or/transports.c index f2c604c..34fe679 100644 --- a/src/or/transports.c +++ b/src/or/transports.c @@ -94,6 +94,7 @@ #include "transports.h" #include "util.h" #include "router.h" +#include "statefile.h" static process_environment_t * create_managed_proxy_environment(const managed_proxy_t *mp);

1 0

[tor/master] Remove all remaining spurious or_options_t invocations in confparse.c
by nickm＠torproject.org 13 Sep '12

13 Sep '12

commit 2eb2536c0a570f7c9f9a168755379f9b04b8465d Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 12 17:39:56 2012 -0400 Remove all remaining spurious or_options_t invocations in confparse.c --- src/or/confparse.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/or/confparse.c b/src/or/confparse.c index b4f06ce..203395d 100644 --- a/src/or/confparse.c +++ b/src/or/confparse.c @@ -214,7 +214,7 @@ config_count_options(const config_format_t *fmt) * Called from config_assign_line() and option_reset(). */ static int -config_assign_value(const config_format_t *fmt, or_options_t *options, +config_assign_value(const config_format_t *fmt, void *options, config_line_t *c, char **msg) { int i, ok; @@ -387,7 +387,7 @@ config_assign_value(const config_format_t *fmt, or_options_t *options, /** Mark every linelist in options "fragile", so that fresh assignments * to it will replace old ones. */ static void -config_mark_lists_fragile(const config_format_t *fmt, or_options_t *options) +config_mark_lists_fragile(const config_format_t *fmt, void *options) { int i; tor_assert(fmt); @@ -416,7 +416,7 @@ config_mark_lists_fragile(const config_format_t *fmt, or_options_t *options) * Called from config_assign(). */ static int -config_assign_line(const config_format_t *fmt, or_options_t *options, +config_assign_line(const config_format_t *fmt, void *options, config_line_t *c, int use_defaults, int clear_first, bitarray_t *options_seen, char **msg) { @@ -484,7 +484,7 @@ config_assign_line(const config_format_t *fmt, or_options_t *options, /** Restore the option named key in options to its default value. * Called from config_assign(). */ static void -config_reset_line(const config_format_t *fmt, or_options_t *options, +config_reset_line(const config_format_t *fmt, void *options, const char *key, int use_defaults) { const config_var_t *var; @@ -937,7 +937,7 @@ config_is_same(const config_format_t *fmt, void * config_dup(const config_format_t *fmt, const void *old) { - or_options_t *newopts; + void *newopts; int i; config_line_t *line; @@ -988,7 +988,7 @@ config_dump(const config_format_t *fmt, const void *default_options, int comment_defaults) { smartlist_t *elements; - const or_options_t *defaults = default_options; + const void *defaults = default_options; void *defaults_tmp = NULL; config_line_t *line, *assigned; char *result;

1 0

[tor/master] Split the generic config_fmt_t code into a new confparse.c file
by nickm＠torproject.org 13 Sep '12

13 Sep '12

commit 7627b2c187ca10ed4569d37ce2b12ff95b00ba33 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 12 17:34:50 2012 -0400 Split the generic config_fmt_t code into a new confparse.c file This helps us split up one of our larger files, and sets the stage for refactoring the configuration backend a little --- changes/refactor_config | 3 + src/or/circuitbuild.c | 1 + src/or/config.c | 1338 +---------------------------------------------- src/or/config.h | 4 +- src/or/confparse.c | 1226 +++++++++++++++++++++++++++++++++++++++++++ src/or/confparse.h | 132 +++++ src/or/control.c | 1 + src/or/dirserv.c | 1 + src/or/include.am | 4 +- src/test/test_config.c | 1 + 10 files changed, 1378 insertions(+), 1333 deletions(-) diff --git a/changes/refactor_config b/changes/refactor_config new file mode 100644 index 0000000..8d272e7 --- /dev/null +++ b/changes/refactor_config @@ -0,0 +1,3 @@ + o Code simplification and refactoring: + - Move the generic "config" code into a new file, and have "config.c" + hold only torrc- and state-related code. diff --git a/src/or/circuitbuild.c b/src/or/circuitbuild.c index 4384e29..ae44de0 100644 --- a/src/or/circuitbuild.c +++ b/src/or/circuitbuild.c @@ -16,6 +16,7 @@ #include "circuitlist.h" #include "circuituse.h" #include "config.h" +#include "confparse.h" #include "connection.h" #include "connection_edge.h" #include "connection_or.h" diff --git a/src/or/config.c b/src/or/config.c index 8f79c44..ce495f3 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -19,6 +19,7 @@ #include "connection_edge.h" #include "connection_or.h" #include "control.h" +#include "confparse.h" #include "cpuworker.h" #include "dirserv.h" #include "dirvote.h" @@ -45,48 +46,6 @@ /* From main.c */ extern int quiet_level; -/** Enumeration of types which option values can take */ -typedef enum config_type_t { - CONFIG_TYPE_STRING = 0, /**< An arbitrary string. */ - CONFIG_TYPE_FILENAME, /**< A filename: some prefixes get expanded. */ - CONFIG_TYPE_UINT, /**< A non-negative integer less than MAX_INT */ - CONFIG_TYPE_INT, /**< Any integer. */ - CONFIG_TYPE_PORT, /**< A port from 1...65535, 0 for "not set", or - * "auto". */ - CONFIG_TYPE_INTERVAL, /**< A number of seconds, with optional units*/ - CONFIG_TYPE_MSEC_INTERVAL,/**< A number of milliseconds, with optional - * units */ - CONFIG_TYPE_MEMUNIT, /**< A number of bytes, with optional units*/ - CONFIG_TYPE_DOUBLE, /**< A floating-point value */ - CONFIG_TYPE_BOOL, /**< A boolean value, expressed as 0 or 1. */ - CONFIG_TYPE_AUTOBOOL, /**< A boolean+auto value, expressed 0 for false, - * 1 for true, and -1 for auto */ - CONFIG_TYPE_ISOTIME, /**< An ISO-formatted time relative to GMT. */ - CONFIG_TYPE_CSV, /**< A list of strings, separated by commas and - * optional whitespace. */ - CONFIG_TYPE_LINELIST, /**< Uninterpreted config lines */ - CONFIG_TYPE_LINELIST_S, /**< Uninterpreted, context-sensitive config lines, - * mixed with other keywords. */ - CONFIG_TYPE_LINELIST_V, /**< Catch-all "virtual" option to summarize - * context-sensitive config lines when fetching. - */ - CONFIG_TYPE_ROUTERSET, /**< A list of router names, addrs, and fps, - * parsed into a routerset_t. */ - CONFIG_TYPE_OBSOLETE, /**< Obsolete (ignored) option. */ -} config_type_t; - -/** An abbreviation for a configuration option allowed on the command line. */ -typedef struct config_abbrev_t { - const char *abbreviated; - const char *full; - int commandline_only; - int warn; -} config_abbrev_t; - -/* Handy macro for declaring "In the config file or on the command line, - * you can abbreviate toks as tok". */ -#define PLURAL(tok) { #tok, #tok "s", 0, 0 } - /** A list of abbreviations and aliases to map command-line options, obsolete * option names, or alternative option names, to their current values. */ static config_abbrev_t _option_abbrevs[] = { @@ -145,16 +104,6 @@ static config_abbrev_t _state_abbrevs[] = { { "EntryNodeUnlistedSince", "EntryGuardUnlistedSince", 0, 0 }, { NULL, NULL, 0, 0}, }; -#undef PLURAL - -/** A variable allowed in the configuration file or on the command line. */ -typedef struct config_var_t { - const char *name; /**< The full keyword (case insensitive). */ - config_type_t type; /**< How to interpret the type and turn it into a - * value. */ - off_t var_offset; /**< Offset of the corresponding member of or_options_t. */ - const char *initvalue; /**< String (or null) describing initial value. */ -} config_var_t; /** An entry for config_vars: "The option name has type * CONFIG_TYPE_conftype, and corresponds to @@ -554,61 +503,9 @@ static config_var_t _state_vars[] = { #undef V #undef OBSOLETE -/** Represents an English description of a configuration variable; used when - * generating configuration file comments. */ -typedef struct config_var_description_t { - const char *name; - const char *description; -} config_var_description_t; - -/** Type of a callback to validate whether a given configuration is - * well-formed and consistent. See options_trial_assign() for documentation - * of arguments. */ -typedef int (*validate_fn_t)(void*,void*,int,char**); - -/** Information on the keys, value types, key-to-struct-member mappings, - * variable descriptions, validation functions, and abbreviations for a - * configuration or storage format. */ -typedef struct { - size_t size; /**< Size of the struct that everything gets parsed into. */ - uint32_t magic; /**< Required 'magic value' to make sure we have a struct - * of the right type. */ - off_t magic_offset; /**< Offset of the magic value within the struct. */ - config_abbrev_t *abbrevs; /**< List of abbreviations that we expand when - * parsing this format. */ - config_var_t *vars; /**< List of variables we recognize, their default - * values, and where we stick them in the structure. */ - validate_fn_t validate_fn; /**< Function to validate config. */ - /** If present, extra is a LINELIST variable for unrecognized - * lines. Otherwise, unrecognized lines are an error. */ - config_var_t *extra; -} config_format_t; - -/** Macro: assert that cfg has the right magic field for format - * fmt. */ -#define CHECK(fmt, cfg) STMT_BEGIN \ - tor_assert(fmt && cfg); \ - tor_assert((fmt)->magic == \ - *(uint32_t*)STRUCT_VAR_P(cfg,fmt->magic_offset)); \ - STMT_END - #ifdef _WIN32 static char *get_windows_conf_root(void); #endif -static void config_line_append(config_line_t **lst, - const char *key, const char *val); -static void option_clear(const config_format_t *fmt, or_options_t *options, - const config_var_t *var); -static void option_reset(const config_format_t *fmt, or_options_t *options, - const config_var_t *var, int use_defaults); -static void config_free(const config_format_t *fmt, void *options); -static int config_lines_eq(config_line_t *a, config_line_t *b); -static int config_count_key(const config_line_t *a, const char *key); -static int option_is_same(const config_format_t *fmt, - const or_options_t *o1, const or_options_t *o2, - const char *name); -static or_options_t *options_dup(const config_format_t *fmt, - const or_options_t *old); static int options_validate(or_options_t *old_options, or_options_t *options, int from_setconf, char **msg); @@ -639,18 +536,11 @@ static int check_server_ports(const smartlist_t *ports, static int validate_data_directory(or_options_t *options); static int write_configuration_file(const char *fname, const or_options_t *options); -static config_line_t *get_assigned_option(const config_format_t *fmt, - const void *options, const char *key, - int escape_val); -static void config_init(const config_format_t *fmt, void *options); static int or_state_validate(or_state_t *old_options, or_state_t *options, int from_setconf, char **msg); static int or_state_load(void); static int options_init_logs(or_options_t *options, int validate_only); -static uint64_t config_parse_memunit(const char *s, int *ok); -static int config_parse_msec_interval(const char *s, int *ok); -static int config_parse_interval(const char *s, int *ok); static void init_libevent(const or_options_t *options); static int opt_streq(const char *s1, const char *s2); @@ -717,16 +607,6 @@ get_dirportfrontpage(void) return global_dirfrontpagecontents; } -/** Allocate an empty configuration object of a given format type. */ -static void * -config_new(const config_format_t *fmt) -{ - void *opts = tor_malloc_zero(fmt->size); - *(uint32_t*)STRUCT_VAR_P(opts, fmt->magic_offset) = fmt->magic; - CHECK(fmt, opts); - return opts; -} - /** Return the currently configured options. */ or_options_t * get_options_mutable(void) @@ -777,8 +657,9 @@ set_options(or_options_t *new_val, char **msg) var->type == CONFIG_TYPE_OBSOLETE) { continue; } - if (!option_is_same(&options_format, new_val, old_options, var_name)) { - line = get_assigned_option(&options_format, new_val, var_name, 1); + if (!config_is_same(&options_format, new_val, old_options, var_name)) { + line = config_get_assigned_option(&options_format, new_val, + var_name, 1); if (line) { for (; line; line = line->next) { @@ -1848,42 +1729,6 @@ options_act(const or_options_t *old_options) return 0; } -/* - * Functions to parse config options - */ - -/** If option is an official abbreviation for a longer option, - * return the longer option. Otherwise return option. - * If command_line is set, apply all abbreviations. Otherwise, only - * apply abbreviations that work for the config file and the command line. - * If warn_obsolete is set, warn about deprecated names. */ -static const char * -expand_abbrev(const config_format_t *fmt, const char *option, int command_line, - int warn_obsolete) -{ - int i; - if (! fmt->abbrevs) - return option; - for (i=0; fmt->abbrevs[i].abbreviated; ++i) { - /* Abbreviations are case insensitive. */ - if (!strcasecmp(option,fmt->abbrevs[i].abbreviated) && - (command_line || !fmt->abbrevs[i].commandline_only)) { - if (warn_obsolete && fmt->abbrevs[i].warn) { - log_warn(LD_CONFIG, - "The configuration option '%s' is deprecated; " - "use '%s' instead.", - fmt->abbrevs[i].abbreviated, - fmt->abbrevs[i].full); - } - /* Keep going through the list in case we want to rewrite it more. - * (We could imagine recursing here, but I don't want to get the - * user into an infinite loop if we craft our list wrong.) */ - option = fmt->abbrevs[i].full; - } - } - return option; -} - /** Helper: Read a list of configuration options from the command line. * If successful, put them in *result and return 0, and return * -1 and leave *result alone. */ @@ -1943,7 +1788,7 @@ config_get_commandlines(int argc, char **argv, config_line_t **result) return -1; } - (*new)->key = tor_strdup(expand_abbrev(&options_format, s, 1, 1)); + (*new)->key = tor_strdup(config_expand_abbrev(&options_format, s, 1, 1)); (*new)->value = want_arg ? tor_strdup(argv[i+1]) : tor_strdup(""); (*new)->command = command; (*new)->next = NULL; @@ -1957,444 +1802,6 @@ config_get_commandlines(int argc, char **argv, config_line_t **result) return 0; } -/** Helper: allocate a new configuration option mapping 'key' to 'val', - * append it to *lst. */ -static void -config_line_append(config_line_t **lst, - const char *key, - const char *val) -{ - config_line_t *newline; - - newline = tor_malloc_zero(sizeof(config_line_t)); - newline->key = tor_strdup(key); - newline->value = tor_strdup(val); - newline->next = NULL; - while (*lst) - lst = &((*lst)->next); - - (*lst) = newline; -} - -/** Helper: parse the config string and strdup into key/value - * strings. Set *result to the list, or NULL if parsing the string - * failed. Return 0 on success, -1 on failure. Warn and ignore any - * misformatted lines. - * - * If extended is set, then treat keys beginning with / and with + as - * indicating "clear" and "append" respectively. */ -int -config_get_lines(const char *string, config_line_t **result, int extended) -{ - config_line_t *list = NULL, **next; - char *k, *v; - - next = &list; - do { - k = v = NULL; - string = parse_config_line_from_str(string, &k, &v); - if (!string) { - config_free_lines(list); - tor_free(k); - tor_free(v); - return -1; - } - if (k && v) { - unsigned command = CONFIG_LINE_NORMAL; - if (extended) { - if (k[0] == '+') { - char *k_new = tor_strdup(k+1); - tor_free(k); - k = k_new; - command = CONFIG_LINE_APPEND; - } else if (k[0] == '/') { - char *k_new = tor_strdup(k+1); - tor_free(k); - k = k_new; - tor_free(v); - v = tor_strdup(""); - command = CONFIG_LINE_CLEAR; - } - } - /* This list can get long, so we keep a pointer to the end of it - * rather than using config_line_append over and over and getting - * n^2 performance. */ - *next = tor_malloc_zero(sizeof(config_line_t)); - (*next)->key = k; - (*next)->value = v; - (*next)->next = NULL; - (*next)->command = command; - next = &((*next)->next); - } else { - tor_free(k); - tor_free(v); - } - } while (*string); - - *result = list; - return 0; -} - -/** - * Free all the configuration lines on the linked list front. - */ -void -config_free_lines(config_line_t *front) -{ - config_line_t *tmp; - - while (front) { - tmp = front; - front = tmp->next; - - tor_free(tmp->key); - tor_free(tmp->value); - tor_free(tmp); - } -} - -/** As config_find_option, but return a non-const pointer. */ -static config_var_t * -config_find_option_mutable(config_format_t *fmt, const char *key) -{ - int i; - size_t keylen = strlen(key); - if (!keylen) - return NULL; /* if they say "--" on the command line, it's not an option */ - /* First, check for an exact (case-insensitive) match */ - for (i=0; fmt->vars[i].name; ++i) { - if (!strcasecmp(key, fmt->vars[i].name)) { - return &fmt->vars[i]; - } - } - /* If none, check for an abbreviated match */ - for (i=0; fmt->vars[i].name; ++i) { - if (!strncasecmp(key, fmt->vars[i].name, keylen)) { - log_warn(LD_CONFIG, "The abbreviation '%s' is deprecated. " - "Please use '%s' instead", - key, fmt->vars[i].name); - return &fmt->vars[i]; - } - } - /* Okay, unrecognized option */ - return NULL; -} - -/** If key is a configuration option, return the corresponding const - * config_var_t. Otherwise, if key is a non-standard abbreviation, - * warn, and return the corresponding const config_var_t. Otherwise return - * NULL. - */ -static const config_var_t * -config_find_option(const config_format_t *fmt, const char *key) -{ - return config_find_option_mutable((config_format_t*)fmt, key); -} - -/** Return the number of option entries in fmt. */ -static int -config_count_options(const config_format_t *fmt) -{ - int i; - for (i=0; fmt->vars[i].name; ++i) - ; - return i; -} - -/* - * Functions to assign config options. - */ - -/** c-\>key is known to be a real key. Update options - * with c-\>value and return 0, or return -1 if bad value. - * - * Called from config_assign_line() and option_reset(). - */ -static int -config_assign_value(const config_format_t *fmt, or_options_t *options, - config_line_t *c, char **msg) -{ - int i, ok; - const config_var_t *var; - void *lvalue; - - CHECK(fmt, options); - - var = config_find_option(fmt, c->key); - tor_assert(var); - - lvalue = STRUCT_VAR_P(options, var->var_offset); - - switch (var->type) { - - case CONFIG_TYPE_PORT: - if (!strcasecmp(c->value, "auto")) { - *(int *)lvalue = CFG_AUTO_PORT; - break; - } - /* fall through */ - case CONFIG_TYPE_INT: - case CONFIG_TYPE_UINT: - i = (int)tor_parse_long(c->value, 10, - var->type==CONFIG_TYPE_INT ? INT_MIN : 0, - var->type==CONFIG_TYPE_PORT ? 65535 : INT_MAX, - &ok, NULL); - if (!ok) { - tor_asprintf(msg, - "Int keyword '%s %s' is malformed or out of bounds.", - c->key, c->value); - return -1; - } - *(int *)lvalue = i; - break; - - case CONFIG_TYPE_INTERVAL: { - i = config_parse_interval(c->value, &ok); - if (!ok) { - tor_asprintf(msg, - "Interval '%s %s' is malformed or out of bounds.", - c->key, c->value); - return -1; - } - *(int *)lvalue = i; - break; - } - - case CONFIG_TYPE_MSEC_INTERVAL: { - i = config_parse_msec_interval(c->value, &ok); - if (!ok) { - tor_asprintf(msg, - "Msec interval '%s %s' is malformed or out of bounds.", - c->key, c->value); - return -1; - } - *(int *)lvalue = i; - break; - } - - case CONFIG_TYPE_MEMUNIT: { - uint64_t u64 = config_parse_memunit(c->value, &ok); - if (!ok) { - tor_asprintf(msg, - "Value '%s %s' is malformed or out of bounds.", - c->key, c->value); - return -1; - } - *(uint64_t *)lvalue = u64; - break; - } - - case CONFIG_TYPE_BOOL: - i = (int)tor_parse_long(c->value, 10, 0, 1, &ok, NULL); - if (!ok) { - tor_asprintf(msg, - "Boolean '%s %s' expects 0 or 1.", - c->key, c->value); - return -1; - } - *(int *)lvalue = i; - break; - - case CONFIG_TYPE_AUTOBOOL: - if (!strcmp(c->value, "auto")) - *(int *)lvalue = -1; - else if (!strcmp(c->value, "0")) - *(int *)lvalue = 0; - else if (!strcmp(c->value, "1")) - *(int *)lvalue = 1; - else { - tor_asprintf(msg, "Boolean '%s %s' expects 0, 1, or 'auto'.", - c->key, c->value); - return -1; - } - break; - - case CONFIG_TYPE_STRING: - case CONFIG_TYPE_FILENAME: - tor_free(*(char **)lvalue); - *(char **)lvalue = tor_strdup(c->value); - break; - - case CONFIG_TYPE_DOUBLE: - *(double *)lvalue = atof(c->value); - break; - - case CONFIG_TYPE_ISOTIME: - if (parse_iso_time(c->value, (time_t *)lvalue)) { - tor_asprintf(msg, - "Invalid time '%s' for keyword '%s'", c->value, c->key); - return -1; - } - break; - - case CONFIG_TYPE_ROUTERSET: - if (*(routerset_t**)lvalue) { - routerset_free(*(routerset_t**)lvalue); - } - *(routerset_t**)lvalue = routerset_new(); - if (routerset_parse(*(routerset_t**)lvalue, c->value, c->key)<0) { - tor_asprintf(msg, "Invalid exit list '%s' for option '%s'", - c->value, c->key); - return -1; - } - break; - - case CONFIG_TYPE_CSV: - if (*(smartlist_t**)lvalue) { - SMARTLIST_FOREACH(*(smartlist_t**)lvalue, char *, cp, tor_free(cp)); - smartlist_clear(*(smartlist_t**)lvalue); - } else { - *(smartlist_t**)lvalue = smartlist_new(); - } - - smartlist_split_string(*(smartlist_t**)lvalue, c->value, ",", - SPLIT_SKIP_SPACE|SPLIT_IGNORE_BLANK, 0); - break; - - case CONFIG_TYPE_LINELIST: - case CONFIG_TYPE_LINELIST_S: - { - config_line_t *lastval = *(config_line_t**)lvalue; - if (lastval && lastval->fragile) { - if (c->command != CONFIG_LINE_APPEND) { - config_free_lines(lastval); - *(config_line_t**)lvalue = NULL; - } else { - lastval->fragile = 0; - } - } - - config_line_append((config_line_t**)lvalue, c->key, c->value); - } - break; - case CONFIG_TYPE_OBSOLETE: - log_warn(LD_CONFIG, "Skipping obsolete configuration option '%s'", c->key); - break; - case CONFIG_TYPE_LINELIST_V: - tor_asprintf(msg, - "You may not provide a value for virtual option '%s'", c->key); - return -1; - default: - tor_assert(0); - break; - } - return 0; -} - -/** Mark every linelist in options "fragile", so that fresh assignments - * to it will replace old ones. */ -static void -config_mark_lists_fragile(const config_format_t *fmt, or_options_t *options) -{ - int i; - tor_assert(fmt); - tor_assert(options); - - for (i = 0; fmt->vars[i].name; ++i) { - const config_var_t *var = &fmt->vars[i]; - config_line_t *list; - if (var->type != CONFIG_TYPE_LINELIST && - var->type != CONFIG_TYPE_LINELIST_V) - continue; - - list = *(config_line_t **)STRUCT_VAR_P(options, var->var_offset); - if (list) - list->fragile = 1; - } -} - -/** If c is a syntactically valid configuration line, update - * options with its value and return 0. Otherwise return -1 for bad - * key, -2 for bad value. - * - * If clear_first is set, clear the value first. Then if - * use_defaults is set, set the value to the default. - * - * Called from config_assign(). - */ -static int -config_assign_line(const config_format_t *fmt, or_options_t *options, - config_line_t *c, int use_defaults, - int clear_first, bitarray_t *options_seen, char **msg) -{ - const config_var_t *var; - - CHECK(fmt, options); - - var = config_find_option(fmt, c->key); - if (!var) { - if (fmt->extra) { - void *lvalue = STRUCT_VAR_P(options, fmt->extra->var_offset); - log_info(LD_CONFIG, - "Found unrecognized option '%s'; saving it.", c->key); - config_line_append((config_line_t**)lvalue, c->key, c->value); - return 0; - } else { - tor_asprintf(msg, - "Unknown option '%s'. Failing.", c->key); - return -1; - } - } - - /* Put keyword into canonical case. */ - if (strcmp(var->name, c->key)) { - tor_free(c->key); - c->key = tor_strdup(var->name); - } - - if (!strlen(c->value)) { - /* reset or clear it, then return */ - if (!clear_first) { - if ((var->type == CONFIG_TYPE_LINELIST || - var->type == CONFIG_TYPE_LINELIST_S) && - c->command != CONFIG_LINE_CLEAR) { - /* We got an empty linelist from the torrc or command line. - As a special case, call this an error. Warn and ignore. */ - log_warn(LD_CONFIG, - "Linelist option '%s' has no value. Skipping.", c->key); - } else { /* not already cleared */ - option_reset(fmt, options, var, use_defaults); - } - } - return 0; - } else if (c->command == CONFIG_LINE_CLEAR && !clear_first) { - option_reset(fmt, options, var, use_defaults); - } - - if (options_seen && (var->type != CONFIG_TYPE_LINELIST && - var->type != CONFIG_TYPE_LINELIST_S)) { - /* We're tracking which options we've seen, and this option is not - * supposed to occur more than once. */ - int var_index = (int)(var - fmt->vars); - if (bitarray_is_set(options_seen, var_index)) { - log_warn(LD_CONFIG, "Option '%s' used more than once; all but the last " - "value will be ignored.", var->name); - } - bitarray_set(options_seen, var_index); - } - - if (config_assign_value(fmt, options, c, msg) < 0) - return -2; - return 0; -} - -/** Restore the option named key in options to its default value. - * Called from config_assign(). */ -static void -config_reset_line(const config_format_t *fmt, or_options_t *options, - const char *key, int use_defaults) -{ - const config_var_t *var; - - CHECK(fmt, options); - - var = config_find_option(fmt, key); - if (!var) - return; /* give error on next pass. */ - - option_reset(fmt, options, var, use_defaults); -} - /** Return true iff key is a valid configuration option. */ int option_is_recognized(const char *key) @@ -2417,287 +1824,7 @@ option_get_canonical_name(const char *key) config_line_t * option_get_assignment(const or_options_t *options, const char *key) { - return get_assigned_option(&options_format, options, key, 1); -} - -/** Return true iff value needs to be quoted and escaped to be used in - * a configuration file. */ -static int -config_value_needs_escape(const char *value) -{ - if (*value == '\"') - return 1; - while (*value) { - switch (*value) - { - case '\r': - case '\n': - case '#': - /* Note: quotes and backspaces need special handling when we are using - * quotes, not otherwise, so they don't trigger escaping on their - * own. */ - return 1; - default: - if (!TOR_ISPRINT(*value)) - return 1; - } - ++value; - } - return 0; -} - -/** Return a newly allocated deep copy of the lines in inp. */ -static config_line_t * -config_lines_dup(const config_line_t *inp) -{ - config_line_t *result = NULL; - config_line_t **next_out = &result; - while (inp) { - *next_out = tor_malloc_zero(sizeof(config_line_t)); - (*next_out)->key = tor_strdup(inp->key); - (*next_out)->value = tor_strdup(inp->value); - inp = inp->next; - next_out = &((*next_out)->next); - } - (*next_out) = NULL; - return result; -} - -/** Return newly allocated line or lines corresponding to key in the - * configuration options. If escape_val is true and a - * value needs to be quoted before it's put in a config file, quote and - * escape that value. Return NULL if no such key exists. */ -static config_line_t * -get_assigned_option(const config_format_t *fmt, const void *options, - const char *key, int escape_val) -{ - const config_var_t *var; - const void *value; - config_line_t *result; - tor_assert(options && key); - - CHECK(fmt, options); - - var = config_find_option(fmt, key); - if (!var) { - log_warn(LD_CONFIG, "Unknown option '%s'. Failing.", key); - return NULL; - } - value = STRUCT_VAR_P(options, var->var_offset); - - result = tor_malloc_zero(sizeof(config_line_t)); - result->key = tor_strdup(var->name); - switch (var->type) - { - case CONFIG_TYPE_STRING: - case CONFIG_TYPE_FILENAME: - if (*(char**)value) { - result->value = tor_strdup(*(char**)value); - } else { - tor_free(result->key); - tor_free(result); - return NULL; - } - break; - case CONFIG_TYPE_ISOTIME: - if (*(time_t*)value) { - result->value = tor_malloc(ISO_TIME_LEN+1); - format_iso_time(result->value, *(time_t*)value); - } else { - tor_free(result->key); - tor_free(result); - } - escape_val = 0; /* Can't need escape. */ - break; - case CONFIG_TYPE_PORT: - if (*(int*)value == CFG_AUTO_PORT) { - result->value = tor_strdup("auto"); - escape_val = 0; - break; - } - /* fall through */ - case CONFIG_TYPE_INTERVAL: - case CONFIG_TYPE_MSEC_INTERVAL: - case CONFIG_TYPE_UINT: - case CONFIG_TYPE_INT: - /* This means every or_options_t uint or bool element - * needs to be an int. Not, say, a uint16_t or char. */ - tor_asprintf(&result->value, "%d", *(int*)value); - escape_val = 0; /* Can't need escape. */ - break; - case CONFIG_TYPE_MEMUNIT: - tor_asprintf(&result->value, U64_FORMAT, - U64_PRINTF_ARG(*(uint64_t*)value)); - escape_val = 0; /* Can't need escape. */ - break; - case CONFIG_TYPE_DOUBLE: - tor_asprintf(&result->value, "%f", *(double*)value); - escape_val = 0; /* Can't need escape. */ - break; - - case CONFIG_TYPE_AUTOBOOL: - if (*(int*)value == -1) { - result->value = tor_strdup("auto"); - escape_val = 0; - break; - } - /* fall through */ - case CONFIG_TYPE_BOOL: - result->value = tor_strdup(*(int*)value ? "1" : "0"); - escape_val = 0; /* Can't need escape. */ - break; - case CONFIG_TYPE_ROUTERSET: - result->value = routerset_to_string(*(routerset_t**)value); - break; - case CONFIG_TYPE_CSV: - if (*(smartlist_t**)value) - result->value = - smartlist_join_strings(*(smartlist_t**)value, ",", 0, NULL); - else - result->value = tor_strdup(""); - break; - case CONFIG_TYPE_OBSOLETE: - log_fn(LOG_PROTOCOL_WARN, LD_CONFIG, - "You asked me for the value of an obsolete config option '%s'.", - key); - tor_free(result->key); - tor_free(result); - return NULL; - case CONFIG_TYPE_LINELIST_S: - log_warn(LD_CONFIG, - "Can't return context-sensitive '%s' on its own", key); - tor_free(result->key); - tor_free(result); - return NULL; - case CONFIG_TYPE_LINELIST: - case CONFIG_TYPE_LINELIST_V: - tor_free(result->key); - tor_free(result); - result = config_lines_dup(*(const config_line_t**)value); - break; - default: - tor_free(result->key); - tor_free(result); - log_warn(LD_BUG,"Unknown type %d for known key '%s'", - var->type, key); - return NULL; - } - - if (escape_val) { - config_line_t *line; - for (line = result; line; line = line->next) { - if (line->value && config_value_needs_escape(line->value)) { - char *newval = esc_for_log(line->value); - tor_free(line->value); - line->value = newval; - } - } - } - - return result; -} - -/** Iterate through the linked list of requested options list. - * For each item, convert as appropriate and assign to options. - * If an item is unrecognized, set *msg and return -1 immediately, - * else return 0 for success. - * - * If clear_first, interpret config options as replacing (not - * extending) their previous values. If clear_first is set, - * then use_defaults to decide if you set to defaults after - * clearing, or make the value 0 or NULL. - * - * Here are the use cases: - * 1. A non-empty AllowInvalid line in your torrc. Appends to current - * if linelist, replaces current if csv. - * 2. An empty AllowInvalid line in your torrc. Should clear it. - * 3. "RESETCONF AllowInvalid" sets it to default. - * 4. "SETCONF AllowInvalid" makes it NULL. - * 5. "SETCONF AllowInvalid=foo" clears it and sets it to "foo". - * - * Use_defaults Clear_first - * 0 0 "append" - * 1 0 undefined, don't use - * 0 1 "set to null first" - * 1 1 "set to defaults first" - * Return 0 on success, -1 on bad key, -2 on bad value. - * - * As an additional special case, if a LINELIST config option has - * no value and clear_first is 0, then warn and ignore it. - */ - -/* -There are three call cases for config_assign() currently. - -Case one: Torrc entry -options_init_from_torrc() calls config_assign(0, 0) - calls config_assign_line(0, 0). - if value is empty, calls option_reset(0) and returns. - calls config_assign_value(), appends. - -Case two: setconf -options_trial_assign() calls config_assign(0, 1) - calls config_reset_line(0) - calls option_reset(0) - calls option_clear(). - calls config_assign_line(0, 1). - if value is empty, returns. - calls config_assign_value(), appends. - -Case three: resetconf -options_trial_assign() calls config_assign(1, 1) - calls config_reset_line(1) - calls option_reset(1) - calls option_clear(). - calls config_assign_value(default) - calls config_assign_line(1, 1). - returns. -*/ -static int -config_assign(const config_format_t *fmt, void *options, config_line_t *list, - int use_defaults, int clear_first, char **msg) -{ - config_line_t *p; - bitarray_t *options_seen; - const int n_options = config_count_options(fmt); - - CHECK(fmt, options); - - /* pass 1: normalize keys */ - for (p = list; p; p = p->next) { - const char *full = expand_abbrev(fmt, p->key, 0, 1); - if (strcmp(full,p->key)) { - tor_free(p->key); - p->key = tor_strdup(full); - } - } - - /* pass 2: if we're reading from a resetting source, clear all - * mentioned config options, and maybe set to their defaults. */ - if (clear_first) { - for (p = list; p; p = p->next) - config_reset_line(fmt, options, p->key, use_defaults); - } - - options_seen = bitarray_init_zero(n_options); - /* pass 3: assign. */ - while (list) { - int r; - if ((r=config_assign_line(fmt, options, list, use_defaults, - clear_first, options_seen, msg))) { - bitarray_free(options_seen); - return r; - } - list = list->next; - } - bitarray_free(options_seen); - - /** Now we're done assigning a group of options to the configuration. - * Subsequent group assignments should _replace_ linelists, not extend - * them. */ - config_mark_lists_fragile(fmt, options); - - return 0; + return config_get_assigned_option(&options_format, options, key, 1); } /** Try assigning list to the global options. You do this by duping @@ -2714,7 +1841,7 @@ options_trial_assign(config_line_t *list, int use_defaults, int clear_first, char **msg) { int r; - or_options_t *trial_options = options_dup(&options_format, get_options()); + or_options_t *trial_options = config_dup(&options_format, get_options()); if ((r=config_assign(&options_format, trial_options, list, use_defaults, clear_first, msg)) < 0) { @@ -2741,90 +1868,6 @@ options_trial_assign(config_line_t *list, int use_defaults, return SETOPT_OK; } -/** Reset config option var to 0, 0.0, NULL, or the equivalent. - * Called from option_reset() and config_free(). */ -static void -option_clear(const config_format_t *fmt, or_options_t *options, - const config_var_t *var) -{ - void *lvalue = STRUCT_VAR_P(options, var->var_offset); - (void)fmt; /* unused */ - switch (var->type) { - case CONFIG_TYPE_STRING: - case CONFIG_TYPE_FILENAME: - tor_free(*(char**)lvalue); - break; - case CONFIG_TYPE_DOUBLE: - *(double*)lvalue = 0.0; - break; - case CONFIG_TYPE_ISOTIME: - *(time_t*)lvalue = 0; - break; - case CONFIG_TYPE_INTERVAL: - case CONFIG_TYPE_MSEC_INTERVAL: - case CONFIG_TYPE_UINT: - case CONFIG_TYPE_INT: - case CONFIG_TYPE_PORT: - case CONFIG_TYPE_BOOL: - *(int*)lvalue = 0; - break; - case CONFIG_TYPE_AUTOBOOL: - *(int*)lvalue = -1; - break; - case CONFIG_TYPE_MEMUNIT: - *(uint64_t*)lvalue = 0; - break; - case CONFIG_TYPE_ROUTERSET: - if (*(routerset_t**)lvalue) { - routerset_free(*(routerset_t**)lvalue); - *(routerset_t**)lvalue = NULL; - } - break; - case CONFIG_TYPE_CSV: - if (*(smartlist_t**)lvalue) { - SMARTLIST_FOREACH(*(smartlist_t **)lvalue, char *, cp, tor_free(cp)); - smartlist_free(*(smartlist_t **)lvalue); - *(smartlist_t **)lvalue = NULL; - } - break; - case CONFIG_TYPE_LINELIST: - case CONFIG_TYPE_LINELIST_S: - config_free_lines(*(config_line_t **)lvalue); - *(config_line_t **)lvalue = NULL; - break; - case CONFIG_TYPE_LINELIST_V: - /* handled by linelist_s. */ - break; - case CONFIG_TYPE_OBSOLETE: - break; - } -} - -/** Clear the option indexed by var in options. Then if - * use_defaults, set it to its default value. - * Called by config_init() and option_reset_line() and option_assign_line(). */ -static void -option_reset(const config_format_t *fmt, or_options_t *options, - const config_var_t *var, int use_defaults) -{ - config_line_t *c; - char *msg = NULL; - CHECK(fmt, options); - option_clear(fmt, options, var); /* clear it first */ - if (!use_defaults) - return; /* all done */ - if (var->initvalue) { - c = tor_malloc_zero(sizeof(config_line_t)); - c->key = tor_strdup(var->name); - c->value = tor_strdup(var->initvalue); - if (config_assign_value(fmt, options, c, &msg) < 0) { - log_warn(LD_BUG, "Failed to assign default: %s", msg); - tor_free(msg); /* if this happens it's a bug */ - } - config_free_lines(c); - } -} - /** Print a usage message for tor. */ static void print_usage(void) @@ -3039,107 +2082,6 @@ is_local_addr(const tor_addr_t *addr) return 0; } -/** Release storage held by options. */ -static void -config_free(const config_format_t *fmt, void *options) -{ - int i; - - if (!options) - return; - - tor_assert(fmt); - - for (i=0; fmt->vars[i].name; ++i) - option_clear(fmt, options, &(fmt->vars[i])); - if (fmt->extra) { - config_line_t **linep = STRUCT_VAR_P(options, fmt->extra->var_offset); - config_free_lines(*linep); - *linep = NULL; - } - tor_free(options); -} - -/** Return true iff a and b contain identical keys and values in identical - * order. */ -static int -config_lines_eq(config_line_t *a, config_line_t *b) -{ - while (a && b) { - if (strcasecmp(a->key, b->key) || strcmp(a->value, b->value)) - return 0; - a = a->next; - b = b->next; - } - if (a || b) - return 0; - return 1; -} - -/** Return the number of lines in a whose key is key. */ -static int -config_count_key(const config_line_t *a, const char *key) -{ - int n = 0; - while (a) { - if (!strcasecmp(a->key, key)) { - ++n; - } - a = a->next; - } - return n; -} - -/** Return true iff the option name has the same value in o1 - * and o2. Must not be called for LINELIST_S or OBSOLETE options. - */ -static int -option_is_same(const config_format_t *fmt, - const or_options_t *o1, const or_options_t *o2, - const char *name) -{ - config_line_t *c1, *c2; - int r = 1; - CHECK(fmt, o1); - CHECK(fmt, o2); - - c1 = get_assigned_option(fmt, o1, name, 0); - c2 = get_assigned_option(fmt, o2, name, 0); - r = config_lines_eq(c1, c2); - config_free_lines(c1); - config_free_lines(c2); - return r; -} - -/** Copy storage held by old into a new or_options_t and return it. */ -static or_options_t * -options_dup(const config_format_t *fmt, const or_options_t *old) -{ - or_options_t *newopts; - int i; - config_line_t *line; - - newopts = config_new(fmt); - for (i=0; fmt->vars[i].name; ++i) { - if (fmt->vars[i].type == CONFIG_TYPE_LINELIST_S) - continue; - if (fmt->vars[i].type == CONFIG_TYPE_OBSOLETE) - continue; - line = get_assigned_option(fmt, old, fmt->vars[i].name, 0); - if (line) { - char *msg = NULL; - if (config_assign(fmt, newopts, line, 0, 0, &msg) < 0) { - log_err(LD_BUG, "Config_get_assigned_option() generated " - "something we couldn't config_assign(): %s", msg); - tor_free(msg); - tor_assert(0); - } - } - config_free_lines(line); - } - return newopts; -} - /** Return a new empty or_options_t. Used for testing. */ or_options_t * options_new(void) @@ -3155,94 +2097,6 @@ options_init(or_options_t *options) config_init(&options_format, options); } -/** Set all vars in the configuration object options to their default - * values. */ -static void -config_init(const config_format_t *fmt, void *options) -{ - int i; - const config_var_t *var; - CHECK(fmt, options); - - for (i=0; fmt->vars[i].name; ++i) { - var = &fmt->vars[i]; - if (!var->initvalue) - continue; /* defaults to NULL or 0 */ - option_reset(fmt, options, var, 1); - } -} - -/** Allocate and return a new string holding the written-out values of the vars - * in 'options'. If 'minimal', do not write out any default-valued vars. - * Else, if comment_defaults, write default values as comments. - */ -static char * -config_dump(const config_format_t *fmt, const void *default_options, - const void *options, int minimal, - int comment_defaults) -{ - smartlist_t *elements; - const or_options_t *defaults = default_options; - void *defaults_tmp = NULL; - config_line_t *line, *assigned; - char *result; - int i; - char *msg = NULL; - - if (defaults == NULL) { - defaults = defaults_tmp = config_new(fmt); - config_init(fmt, defaults_tmp); - } - - /* XXX use a 1 here so we don't add a new log line while dumping */ - if (default_options == NULL) { - if (fmt->validate_fn(NULL, defaults_tmp, 1, &msg) < 0) { - log_err(LD_BUG, "Failed to validate default config."); - tor_free(msg); - tor_assert(0); - } - } - - elements = smartlist_new(); - for (i=0; fmt->vars[i].name; ++i) { - int comment_option = 0; - if (fmt->vars[i].type == CONFIG_TYPE_OBSOLETE || - fmt->vars[i].type == CONFIG_TYPE_LINELIST_S) - continue; - /* Don't save 'hidden' control variables. */ - if (!strcmpstart(fmt->vars[i].name, "__")) - continue; - if (minimal && option_is_same(fmt, options, defaults, fmt->vars[i].name)) - continue; - else if (comment_defaults && - option_is_same(fmt, options, defaults, fmt->vars[i].name)) - comment_option = 1; - - line = assigned = get_assigned_option(fmt, options, fmt->vars[i].name, 1); - - for (; line; line = line->next) { - smartlist_add_asprintf(elements, "%s%s %s\n", - comment_option ? "# " : "", - line->key, line->value); - } - config_free_lines(assigned); - } - - if (fmt->extra) { - line = *(config_line_t**)STRUCT_VAR_P(options, fmt->extra->var_offset); - for (; line; line = line->next) { - smartlist_add_asprintf(elements, "%s %s\n", line->key, line->value); - } - } - - result = smartlist_join_strings(elements, "", 0, NULL); - SMARTLIST_FOREACH(elements, char *, cp, tor_free(cp)); - smartlist_free(elements); - if (defaults_tmp) - config_free(fmt, defaults_tmp); - return result; -} - /** Return a string containing a possible configuration file that would give * the configuration in options. If minimal is true, do not * include options that are the same as Tor's defaults. @@ -4753,7 +3607,7 @@ options_init_from_string(const char *cf_defaults, const char *cf, goto err; } if (i==0) - newdefaultoptions = options_dup(&options_format, newoptions); + newdefaultoptions = config_dup(&options_format, newoptions); } /* Go through command-line variables too */ @@ -4814,7 +3668,7 @@ options_init_from_string(const char *cf_defaults, const char *cf, goto err; } if (i==0) - newdefaultoptions = options_dup(&options_format, newoptions); + newdefaultoptions = config_dup(&options_format, newoptions); } /* Assign command-line variables a second time too */ retval = config_assign(&options_format, newoptions, @@ -6484,180 +5338,6 @@ options_save_current(void) return write_configuration_file(get_torrc_fname(0), get_options()); } -/** Mapping from a unit name to a multiplier for converting that unit into a - * base unit. Used by config_parse_unit. */ -struct unit_table_t { - const char *unit; /**< The name of the unit */ - uint64_t multiplier; /**< How many of the base unit appear in this unit */ -}; - -/** Table to map the names of memory units to the number of bytes they - * contain. */ -static struct unit_table_t memory_units[] = { - { "", 1 }, - { "b", 1<< 0 }, - { "byte", 1<< 0 }, - { "bytes", 1<< 0 }, - { "kb", 1<<10 }, - { "kbyte", 1<<10 }, - { "kbytes", 1<<10 }, - { "kilobyte", 1<<10 }, - { "kilobytes", 1<<10 }, - { "m", 1<<20 }, - { "mb", 1<<20 }, - { "mbyte", 1<<20 }, - { "mbytes", 1<<20 }, - { "megabyte", 1<<20 }, - { "megabytes", 1<<20 }, - { "gb", 1<<30 }, - { "gbyte", 1<<30 }, - { "gbytes", 1<<30 }, - { "gigabyte", 1<<30 }, - { "gigabytes", 1<<30 }, - { "tb", U64_LITERAL(1)<<40 }, - { "terabyte", U64_LITERAL(1)<<40 }, - { "terabytes", U64_LITERAL(1)<<40 }, - { NULL, 0 }, -}; - -/** Table to map the names of time units to the number of seconds they - * contain. */ -static struct unit_table_t time_units[] = { - { "", 1 }, - { "second", 1 }, - { "seconds", 1 }, - { "minute", 60 }, - { "minutes", 60 }, - { "hour", 60*60 }, - { "hours", 60*60 }, - { "day", 24*60*60 }, - { "days", 24*60*60 }, - { "week", 7*24*60*60 }, - { "weeks", 7*24*60*60 }, - { NULL, 0 }, -}; - -/** Table to map the names of time units to the number of milliseconds - * they contain. */ -static struct unit_table_t time_msec_units[] = { - { "", 1 }, - { "msec", 1 }, - { "millisecond", 1 }, - { "milliseconds", 1 }, - { "second", 1000 }, - { "seconds", 1000 }, - { "minute", 60*1000 }, - { "minutes", 60*1000 }, - { "hour", 60*60*1000 }, - { "hours", 60*60*1000 }, - { "day", 24*60*60*1000 }, - { "days", 24*60*60*1000 }, - { "week", 7*24*60*60*1000 }, - { "weeks", 7*24*60*60*1000 }, - { NULL, 0 }, -}; - -/** Parse a string val containing a number, zero or more - * spaces, and an optional unit string. If the unit appears in the - * table u, then multiply the number by the unit multiplier. - * On success, set *ok to 1 and return this product. - * Otherwise, set *ok to 0. - */ -static uint64_t -config_parse_units(const char *val, struct unit_table_t *u, int *ok) -{ - uint64_t v = 0; - double d = 0; - int use_float = 0; - char *cp; - - tor_assert(ok); - - v = tor_parse_uint64(val, 10, 0, UINT64_MAX, ok, &cp); - if (!*ok || (cp && *cp == '.')) { - d = tor_parse_double(val, 0, UINT64_MAX, ok, &cp); - if (!*ok) - goto done; - use_float = 1; - } - - if (!cp) { - *ok = 1; - v = use_float ? DBL_TO_U64(d) : v; - goto done; - } - - cp = (char*) eat_whitespace(cp); - - for ( ;u->unit;++u) { - if (!strcasecmp(u->unit, cp)) { - if (use_float) - v = u->multiplier * d; - else - v *= u->multiplier; - *ok = 1; - goto done; - } - } - log_warn(LD_CONFIG, "Unknown unit '%s'.", cp); - *ok = 0; - done: - - if (*ok) - return v; - else - return 0; -} - -/** Parse a string in the format "number unit", where unit is a unit of - * information (byte, KB, M, etc). On success, set *ok to true - * and return the number of bytes specified. Otherwise, set - * *ok to false and return 0. */ -static uint64_t -config_parse_memunit(const char *s, int *ok) -{ - uint64_t u = config_parse_units(s, memory_units, ok); - return u; -} - -/** Parse a string in the format "number unit", where unit is a unit of - * time in milliseconds. On success, set *ok to true and return - * the number of milliseconds in the provided interval. Otherwise, set - * *ok to 0 and return -1. */ -static int -config_parse_msec_interval(const char *s, int *ok) -{ - uint64_t r; - r = config_parse_units(s, time_msec_units, ok); - if (!ok) - return -1; - if (r > INT_MAX) { - log_warn(LD_CONFIG, "Msec interval '%s' is too long", s); - *ok = 0; - return -1; - } - return (int)r; -} - -/** Parse a string in the format "number unit", where unit is a unit of time. - * On success, set *ok to true and return the number of seconds in - * the provided interval. Otherwise, set *ok to 0 and return -1. - */ -static int -config_parse_interval(const char *s, int *ok) -{ - uint64_t r; - r = config_parse_units(s, time_units, ok); - if (!ok) - return -1; - if (r > INT_MAX) { - log_warn(LD_CONFIG, "Interval '%s' is too long", s); - *ok = 0; - return -1; - } - return (int)r; -} - /** Return the number of cpus configured in options. If we are * told to auto-detect the number of cpus, return the auto-detected number. */ int diff --git a/src/or/config.h b/src/or/config.h index d207965..abfbef7 100644 --- a/src/or/config.h +++ b/src/or/config.h @@ -23,11 +23,9 @@ const char *escaped_safe_str_client(const char *address); const char *escaped_safe_str(const char *address); const char *get_version(void); const char *get_short_version(void); - -int config_get_lines(const char *string, config_line_t **result, int extended); -void config_free_lines(config_line_t *front); setopt_err_t options_trial_assign(config_line_t *list, int use_defaults, int clear_first, char **msg); + int resolve_my_address(int warn_severity, const or_options_t *options, uint32_t *addr, char **hostname_out); int is_local_addr(const tor_addr_t *addr); diff --git a/src/or/confparse.c b/src/or/confparse.c new file mode 100644 index 0000000..b4f06ce --- /dev/null +++ b/src/or/confparse.c @@ -0,0 +1,1226 @@ +/* Copyright (c) 2001 Matej Pfajfar. + * Copyright (c) 2001-2004, Roger Dingledine. + * Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson. + * Copyright (c) 2007-2012, The Tor Project, Inc. */ +/* See LICENSE for licensing information */ + +#include "or.h" +#include "confparse.h" +#include "routerlist.h" + +static uint64_t config_parse_memunit(const char *s, int *ok); +static int config_parse_msec_interval(const char *s, int *ok); +static int config_parse_interval(const char *s, int *ok); +static void config_reset(const config_format_t *fmt, void *options, + const config_var_t *var, int use_defaults); + +/** Allocate an empty configuration object of a given format type. */ +void * +config_new(const config_format_t *fmt) +{ + void *opts = tor_malloc_zero(fmt->size); + *(uint32_t*)STRUCT_VAR_P(opts, fmt->magic_offset) = fmt->magic; + CONFIG_CHECK(fmt, opts); + return opts; +} + +/* + * Functions to parse config options + */ + +/** If option is an official abbreviation for a longer option, + * return the longer option. Otherwise return option. + * If command_line is set, apply all abbreviations. Otherwise, only + * apply abbreviations that work for the config file and the command line. + * If warn_obsolete is set, warn about deprecated names. */ +const char * +config_expand_abbrev(const config_format_t *fmt, const char *option, + int command_line, int warn_obsolete) +{ + int i; + if (! fmt->abbrevs) + return option; + for (i=0; fmt->abbrevs[i].abbreviated; ++i) { + /* Abbreviations are case insensitive. */ + if (!strcasecmp(option,fmt->abbrevs[i].abbreviated) && + (command_line || !fmt->abbrevs[i].commandline_only)) { + if (warn_obsolete && fmt->abbrevs[i].warn) { + log_warn(LD_CONFIG, + "The configuration option '%s' is deprecated; " + "use '%s' instead.", + fmt->abbrevs[i].abbreviated, + fmt->abbrevs[i].full); + } + /* Keep going through the list in case we want to rewrite it more. + * (We could imagine recursing here, but I don't want to get the + * user into an infinite loop if we craft our list wrong.) */ + option = fmt->abbrevs[i].full; + } + } + return option; +} + +/** Helper: allocate a new configuration option mapping 'key' to 'val', + * append it to *lst. */ +void +config_line_append(config_line_t **lst, + const char *key, + const char *val) +{ + config_line_t *newline; + + newline = tor_malloc_zero(sizeof(config_line_t)); + newline->key = tor_strdup(key); + newline->value = tor_strdup(val); + newline->next = NULL; + while (*lst) + lst = &((*lst)->next); + + (*lst) = newline; +} + +/** Helper: parse the config string and strdup into key/value + * strings. Set *result to the list, or NULL if parsing the string + * failed. Return 0 on success, -1 on failure. Warn and ignore any + * misformatted lines. + * + * If extended is set, then treat keys beginning with / and with + as + * indicating "clear" and "append" respectively. */ +int +config_get_lines(const char *string, config_line_t **result, int extended) +{ + config_line_t *list = NULL, **next; + char *k, *v; + + next = &list; + do { + k = v = NULL; + string = parse_config_line_from_str(string, &k, &v); + if (!string) { + config_free_lines(list); + tor_free(k); + tor_free(v); + return -1; + } + if (k && v) { + unsigned command = CONFIG_LINE_NORMAL; + if (extended) { + if (k[0] == '+') { + char *k_new = tor_strdup(k+1); + tor_free(k); + k = k_new; + command = CONFIG_LINE_APPEND; + } else if (k[0] == '/') { + char *k_new = tor_strdup(k+1); + tor_free(k); + k = k_new; + tor_free(v); + v = tor_strdup(""); + command = CONFIG_LINE_CLEAR; + } + } + /* This list can get long, so we keep a pointer to the end of it + * rather than using config_line_append over and over and getting + * n^2 performance. */ + *next = tor_malloc_zero(sizeof(config_line_t)); + (*next)->key = k; + (*next)->value = v; + (*next)->next = NULL; + (*next)->command = command; + next = &((*next)->next); + } else { + tor_free(k); + tor_free(v); + } + } while (*string); + + *result = list; + return 0; +} + +/** + * Free all the configuration lines on the linked list front. + */ +void +config_free_lines(config_line_t *front) +{ + config_line_t *tmp; + + while (front) { + tmp = front; + front = tmp->next; + + tor_free(tmp->key); + tor_free(tmp->value); + tor_free(tmp); + } +} + +/** As config_find_option, but return a non-const pointer. */ +config_var_t * +config_find_option_mutable(config_format_t *fmt, const char *key) +{ + int i; + size_t keylen = strlen(key); + if (!keylen) + return NULL; /* if they say "--" on the command line, it's not an option */ + /* First, check for an exact (case-insensitive) match */ + for (i=0; fmt->vars[i].name; ++i) { + if (!strcasecmp(key, fmt->vars[i].name)) { + return &fmt->vars[i]; + } + } + /* If none, check for an abbreviated match */ + for (i=0; fmt->vars[i].name; ++i) { + if (!strncasecmp(key, fmt->vars[i].name, keylen)) { + log_warn(LD_CONFIG, "The abbreviation '%s' is deprecated. " + "Please use '%s' instead", + key, fmt->vars[i].name); + return &fmt->vars[i]; + } + } + /* Okay, unrecognized option */ + return NULL; +} + +/** If key is a configuration option, return the corresponding const + * config_var_t. Otherwise, if key is a non-standard abbreviation, + * warn, and return the corresponding const config_var_t. Otherwise return + * NULL. + */ +const config_var_t * +config_find_option(const config_format_t *fmt, const char *key) +{ + return config_find_option_mutable((config_format_t*)fmt, key); +} + +/** Return the number of option entries in fmt. */ +static int +config_count_options(const config_format_t *fmt) +{ + int i; + for (i=0; fmt->vars[i].name; ++i) + ; + return i; +} + +/* + * Functions to assign config options. + */ + +/** c-\>key is known to be a real key. Update options + * with c-\>value and return 0, or return -1 if bad value. + * + * Called from config_assign_line() and option_reset(). + */ +static int +config_assign_value(const config_format_t *fmt, or_options_t *options, + config_line_t *c, char **msg) +{ + int i, ok; + const config_var_t *var; + void *lvalue; + + CONFIG_CHECK(fmt, options); + + var = config_find_option(fmt, c->key); + tor_assert(var); + + lvalue = STRUCT_VAR_P(options, var->var_offset); + + switch (var->type) { + + case CONFIG_TYPE_PORT: + if (!strcasecmp(c->value, "auto")) { + *(int *)lvalue = CFG_AUTO_PORT; + break; + } + /* fall through */ + case CONFIG_TYPE_INT: + case CONFIG_TYPE_UINT: + i = (int)tor_parse_long(c->value, 10, + var->type==CONFIG_TYPE_INT ? INT_MIN : 0, + var->type==CONFIG_TYPE_PORT ? 65535 : INT_MAX, + &ok, NULL); + if (!ok) { + tor_asprintf(msg, + "Int keyword '%s %s' is malformed or out of bounds.", + c->key, c->value); + return -1; + } + *(int *)lvalue = i; + break; + + case CONFIG_TYPE_INTERVAL: { + i = config_parse_interval(c->value, &ok); + if (!ok) { + tor_asprintf(msg, + "Interval '%s %s' is malformed or out of bounds.", + c->key, c->value); + return -1; + } + *(int *)lvalue = i; + break; + } + + case CONFIG_TYPE_MSEC_INTERVAL: { + i = config_parse_msec_interval(c->value, &ok); + if (!ok) { + tor_asprintf(msg, + "Msec interval '%s %s' is malformed or out of bounds.", + c->key, c->value); + return -1; + } + *(int *)lvalue = i; + break; + } + + case CONFIG_TYPE_MEMUNIT: { + uint64_t u64 = config_parse_memunit(c->value, &ok); + if (!ok) { + tor_asprintf(msg, + "Value '%s %s' is malformed or out of bounds.", + c->key, c->value); + return -1; + } + *(uint64_t *)lvalue = u64; + break; + } + + case CONFIG_TYPE_BOOL: + i = (int)tor_parse_long(c->value, 10, 0, 1, &ok, NULL); + if (!ok) { + tor_asprintf(msg, + "Boolean '%s %s' expects 0 or 1.", + c->key, c->value); + return -1; + } + *(int *)lvalue = i; + break; + + case CONFIG_TYPE_AUTOBOOL: + if (!strcmp(c->value, "auto")) + *(int *)lvalue = -1; + else if (!strcmp(c->value, "0")) + *(int *)lvalue = 0; + else if (!strcmp(c->value, "1")) + *(int *)lvalue = 1; + else { + tor_asprintf(msg, "Boolean '%s %s' expects 0, 1, or 'auto'.", + c->key, c->value); + return -1; + } + break; + + case CONFIG_TYPE_STRING: + case CONFIG_TYPE_FILENAME: + tor_free(*(char **)lvalue); + *(char **)lvalue = tor_strdup(c->value); + break; + + case CONFIG_TYPE_DOUBLE: + *(double *)lvalue = atof(c->value); + break; + + case CONFIG_TYPE_ISOTIME: + if (parse_iso_time(c->value, (time_t *)lvalue)) { + tor_asprintf(msg, + "Invalid time '%s' for keyword '%s'", c->value, c->key); + return -1; + } + break; + + case CONFIG_TYPE_ROUTERSET: + if (*(routerset_t**)lvalue) { + routerset_free(*(routerset_t**)lvalue); + } + *(routerset_t**)lvalue = routerset_new(); + if (routerset_parse(*(routerset_t**)lvalue, c->value, c->key)<0) { + tor_asprintf(msg, "Invalid exit list '%s' for option '%s'", + c->value, c->key); + return -1; + } + break; + + case CONFIG_TYPE_CSV: + if (*(smartlist_t**)lvalue) { + SMARTLIST_FOREACH(*(smartlist_t**)lvalue, char *, cp, tor_free(cp)); + smartlist_clear(*(smartlist_t**)lvalue); + } else { + *(smartlist_t**)lvalue = smartlist_new(); + } + + smartlist_split_string(*(smartlist_t**)lvalue, c->value, ",", + SPLIT_SKIP_SPACE|SPLIT_IGNORE_BLANK, 0); + break; + + case CONFIG_TYPE_LINELIST: + case CONFIG_TYPE_LINELIST_S: + { + config_line_t *lastval = *(config_line_t**)lvalue; + if (lastval && lastval->fragile) { + if (c->command != CONFIG_LINE_APPEND) { + config_free_lines(lastval); + *(config_line_t**)lvalue = NULL; + } else { + lastval->fragile = 0; + } + } + + config_line_append((config_line_t**)lvalue, c->key, c->value); + } + break; + case CONFIG_TYPE_OBSOLETE: + log_warn(LD_CONFIG, "Skipping obsolete configuration option '%s'", c->key); + break; + case CONFIG_TYPE_LINELIST_V: + tor_asprintf(msg, + "You may not provide a value for virtual option '%s'", c->key); + return -1; + default: + tor_assert(0); + break; + } + return 0; +} + +/** Mark every linelist in options "fragile", so that fresh assignments + * to it will replace old ones. */ +static void +config_mark_lists_fragile(const config_format_t *fmt, or_options_t *options) +{ + int i; + tor_assert(fmt); + tor_assert(options); + + for (i = 0; fmt->vars[i].name; ++i) { + const config_var_t *var = &fmt->vars[i]; + config_line_t *list; + if (var->type != CONFIG_TYPE_LINELIST && + var->type != CONFIG_TYPE_LINELIST_V) + continue; + + list = *(config_line_t **)STRUCT_VAR_P(options, var->var_offset); + if (list) + list->fragile = 1; + } +} + +/** If c is a syntactically valid configuration line, update + * options with its value and return 0. Otherwise return -1 for bad + * key, -2 for bad value. + * + * If clear_first is set, clear the value first. Then if + * use_defaults is set, set the value to the default. + * + * Called from config_assign(). + */ +static int +config_assign_line(const config_format_t *fmt, or_options_t *options, + config_line_t *c, int use_defaults, + int clear_first, bitarray_t *options_seen, char **msg) +{ + const config_var_t *var; + + CONFIG_CHECK(fmt, options); + + var = config_find_option(fmt, c->key); + if (!var) { + if (fmt->extra) { + void *lvalue = STRUCT_VAR_P(options, fmt->extra->var_offset); + log_info(LD_CONFIG, + "Found unrecognized option '%s'; saving it.", c->key); + config_line_append((config_line_t**)lvalue, c->key, c->value); + return 0; + } else { + tor_asprintf(msg, + "Unknown option '%s'. Failing.", c->key); + return -1; + } + } + + /* Put keyword into canonical case. */ + if (strcmp(var->name, c->key)) { + tor_free(c->key); + c->key = tor_strdup(var->name); + } + + if (!strlen(c->value)) { + /* reset or clear it, then return */ + if (!clear_first) { + if ((var->type == CONFIG_TYPE_LINELIST || + var->type == CONFIG_TYPE_LINELIST_S) && + c->command != CONFIG_LINE_CLEAR) { + /* We got an empty linelist from the torrc or command line. + As a special case, call this an error. Warn and ignore. */ + log_warn(LD_CONFIG, + "Linelist option '%s' has no value. Skipping.", c->key); + } else { /* not already cleared */ + config_reset(fmt, options, var, use_defaults); + } + } + return 0; + } else if (c->command == CONFIG_LINE_CLEAR && !clear_first) { + config_reset(fmt, options, var, use_defaults); + } + + if (options_seen && (var->type != CONFIG_TYPE_LINELIST && + var->type != CONFIG_TYPE_LINELIST_S)) { + /* We're tracking which options we've seen, and this option is not + * supposed to occur more than once. */ + int var_index = (int)(var - fmt->vars); + if (bitarray_is_set(options_seen, var_index)) { + log_warn(LD_CONFIG, "Option '%s' used more than once; all but the last " + "value will be ignored.", var->name); + } + bitarray_set(options_seen, var_index); + } + + if (config_assign_value(fmt, options, c, msg) < 0) + return -2; + return 0; +} + +/** Restore the option named key in options to its default value. + * Called from config_assign(). */ +static void +config_reset_line(const config_format_t *fmt, or_options_t *options, + const char *key, int use_defaults) +{ + const config_var_t *var; + + CONFIG_CHECK(fmt, options); + + var = config_find_option(fmt, key); + if (!var) + return; /* give error on next pass. */ + + config_reset(fmt, options, var, use_defaults); +} + +/** Return true iff value needs to be quoted and escaped to be used in + * a configuration file. */ +static int +config_value_needs_escape(const char *value) +{ + if (*value == '\"') + return 1; + while (*value) { + switch (*value) + { + case '\r': + case '\n': + case '#': + /* Note: quotes and backspaces need special handling when we are using + * quotes, not otherwise, so they don't trigger escaping on their + * own. */ + return 1; + default: + if (!TOR_ISPRINT(*value)) + return 1; + } + ++value; + } + return 0; +} + +/** Return a newly allocated deep copy of the lines in inp. */ +config_line_t * +config_lines_dup(const config_line_t *inp) +{ + config_line_t *result = NULL; + config_line_t **next_out = &result; + while (inp) { + *next_out = tor_malloc_zero(sizeof(config_line_t)); + (*next_out)->key = tor_strdup(inp->key); + (*next_out)->value = tor_strdup(inp->value); + inp = inp->next; + next_out = &((*next_out)->next); + } + (*next_out) = NULL; + return result; +} + +/** Return newly allocated line or lines corresponding to key in the + * configuration options. If escape_val is true and a + * value needs to be quoted before it's put in a config file, quote and + * escape that value. Return NULL if no such key exists. */ +config_line_t * +config_get_assigned_option(const config_format_t *fmt, const void *options, + const char *key, int escape_val) +{ + const config_var_t *var; + const void *value; + config_line_t *result; + tor_assert(options && key); + + CONFIG_CHECK(fmt, options); + + var = config_find_option(fmt, key); + if (!var) { + log_warn(LD_CONFIG, "Unknown option '%s'. Failing.", key); + return NULL; + } + value = STRUCT_VAR_P(options, var->var_offset); + + result = tor_malloc_zero(sizeof(config_line_t)); + result->key = tor_strdup(var->name); + switch (var->type) + { + case CONFIG_TYPE_STRING: + case CONFIG_TYPE_FILENAME: + if (*(char**)value) { + result->value = tor_strdup(*(char**)value); + } else { + tor_free(result->key); + tor_free(result); + return NULL; + } + break; + case CONFIG_TYPE_ISOTIME: + if (*(time_t*)value) { + result->value = tor_malloc(ISO_TIME_LEN+1); + format_iso_time(result->value, *(time_t*)value); + } else { + tor_free(result->key); + tor_free(result); + } + escape_val = 0; /* Can't need escape. */ + break; + case CONFIG_TYPE_PORT: + if (*(int*)value == CFG_AUTO_PORT) { + result->value = tor_strdup("auto"); + escape_val = 0; + break; + } + /* fall through */ + case CONFIG_TYPE_INTERVAL: + case CONFIG_TYPE_MSEC_INTERVAL: + case CONFIG_TYPE_UINT: + case CONFIG_TYPE_INT: + /* This means every or_options_t uint or bool element + * needs to be an int. Not, say, a uint16_t or char. */ + tor_asprintf(&result->value, "%d", *(int*)value); + escape_val = 0; /* Can't need escape. */ + break; + case CONFIG_TYPE_MEMUNIT: + tor_asprintf(&result->value, U64_FORMAT, + U64_PRINTF_ARG(*(uint64_t*)value)); + escape_val = 0; /* Can't need escape. */ + break; + case CONFIG_TYPE_DOUBLE: + tor_asprintf(&result->value, "%f", *(double*)value); + escape_val = 0; /* Can't need escape. */ + break; + + case CONFIG_TYPE_AUTOBOOL: + if (*(int*)value == -1) { + result->value = tor_strdup("auto"); + escape_val = 0; + break; + } + /* fall through */ + case CONFIG_TYPE_BOOL: + result->value = tor_strdup(*(int*)value ? "1" : "0"); + escape_val = 0; /* Can't need escape. */ + break; + case CONFIG_TYPE_ROUTERSET: + result->value = routerset_to_string(*(routerset_t**)value); + break; + case CONFIG_TYPE_CSV: + if (*(smartlist_t**)value) + result->value = + smartlist_join_strings(*(smartlist_t**)value, ",", 0, NULL); + else + result->value = tor_strdup(""); + break; + case CONFIG_TYPE_OBSOLETE: + log_fn(LOG_INFO, LD_CONFIG, + "You asked me for the value of an obsolete config option '%s'.", + key); + tor_free(result->key); + tor_free(result); + return NULL; + case CONFIG_TYPE_LINELIST_S: + log_warn(LD_CONFIG, + "Can't return context-sensitive '%s' on its own", key); + tor_free(result->key); + tor_free(result); + return NULL; + case CONFIG_TYPE_LINELIST: + case CONFIG_TYPE_LINELIST_V: + tor_free(result->key); + tor_free(result); + result = config_lines_dup(*(const config_line_t**)value); + break; + default: + tor_free(result->key); + tor_free(result); + log_warn(LD_BUG,"Unknown type %d for known key '%s'", + var->type, key); + return NULL; + } + + if (escape_val) { + config_line_t *line; + for (line = result; line; line = line->next) { + if (line->value && config_value_needs_escape(line->value)) { + char *newval = esc_for_log(line->value); + tor_free(line->value); + line->value = newval; + } + } + } + + return result; +} +/** Iterate through the linked list of requested options list. + * For each item, convert as appropriate and assign to options. + * If an item is unrecognized, set *msg and return -1 immediately, + * else return 0 for success. + * + * If clear_first, interpret config options as replacing (not + * extending) their previous values. If clear_first is set, + * then use_defaults to decide if you set to defaults after + * clearing, or make the value 0 or NULL. + * + * Here are the use cases: + * 1. A non-empty AllowInvalid line in your torrc. Appends to current + * if linelist, replaces current if csv. + * 2. An empty AllowInvalid line in your torrc. Should clear it. + * 3. "RESETCONF AllowInvalid" sets it to default. + * 4. "SETCONF AllowInvalid" makes it NULL. + * 5. "SETCONF AllowInvalid=foo" clears it and sets it to "foo". + * + * Use_defaults Clear_first + * 0 0 "append" + * 1 0 undefined, don't use + * 0 1 "set to null first" + * 1 1 "set to defaults first" + * Return 0 on success, -1 on bad key, -2 on bad value. + * + * As an additional special case, if a LINELIST config option has + * no value and clear_first is 0, then warn and ignore it. + */ + +/* +There are three call cases for config_assign() currently. + +Case one: Torrc entry +options_init_from_torrc() calls config_assign(0, 0) + calls config_assign_line(0, 0). + if value is empty, calls config_reset(0) and returns. + calls config_assign_value(), appends. + +Case two: setconf +options_trial_assign() calls config_assign(0, 1) + calls config_reset_line(0) + calls config_reset(0) + calls option_clear(). + calls config_assign_line(0, 1). + if value is empty, returns. + calls config_assign_value(), appends. + +Case three: resetconf +options_trial_assign() calls config_assign(1, 1) + calls config_reset_line(1) + calls config_reset(1) + calls option_clear(). + calls config_assign_value(default) + calls config_assign_line(1, 1). + returns. +*/ +int +config_assign(const config_format_t *fmt, void *options, config_line_t *list, + int use_defaults, int clear_first, char **msg) +{ + config_line_t *p; + bitarray_t *options_seen; + const int n_options = config_count_options(fmt); + + CONFIG_CHECK(fmt, options); + + /* pass 1: normalize keys */ + for (p = list; p; p = p->next) { + const char *full = config_expand_abbrev(fmt, p->key, 0, 1); + if (strcmp(full,p->key)) { + tor_free(p->key); + p->key = tor_strdup(full); + } + } + + /* pass 2: if we're reading from a resetting source, clear all + * mentioned config options, and maybe set to their defaults. */ + if (clear_first) { + for (p = list; p; p = p->next) + config_reset_line(fmt, options, p->key, use_defaults); + } + + options_seen = bitarray_init_zero(n_options); + /* pass 3: assign. */ + while (list) { + int r; + if ((r=config_assign_line(fmt, options, list, use_defaults, + clear_first, options_seen, msg))) { + bitarray_free(options_seen); + return r; + } + list = list->next; + } + bitarray_free(options_seen); + + /** Now we're done assigning a group of options to the configuration. + * Subsequent group assignments should _replace_ linelists, not extend + * them. */ + config_mark_lists_fragile(fmt, options); + + return 0; +} + +/** Reset config option var to 0, 0.0, NULL, or the equivalent. + * Called from config_reset() and config_free(). */ +static void +config_clear(const config_format_t *fmt, void *options, + const config_var_t *var) +{ + void *lvalue = STRUCT_VAR_P(options, var->var_offset); + (void)fmt; /* unused */ + switch (var->type) { + case CONFIG_TYPE_STRING: + case CONFIG_TYPE_FILENAME: + tor_free(*(char**)lvalue); + break; + case CONFIG_TYPE_DOUBLE: + *(double*)lvalue = 0.0; + break; + case CONFIG_TYPE_ISOTIME: + *(time_t*)lvalue = 0; + break; + case CONFIG_TYPE_INTERVAL: + case CONFIG_TYPE_MSEC_INTERVAL: + case CONFIG_TYPE_UINT: + case CONFIG_TYPE_INT: + case CONFIG_TYPE_PORT: + case CONFIG_TYPE_BOOL: + *(int*)lvalue = 0; + break; + case CONFIG_TYPE_AUTOBOOL: + *(int*)lvalue = -1; + break; + case CONFIG_TYPE_MEMUNIT: + *(uint64_t*)lvalue = 0; + break; + case CONFIG_TYPE_ROUTERSET: + if (*(routerset_t**)lvalue) { + routerset_free(*(routerset_t**)lvalue); + *(routerset_t**)lvalue = NULL; + } + break; + case CONFIG_TYPE_CSV: + if (*(smartlist_t**)lvalue) { + SMARTLIST_FOREACH(*(smartlist_t **)lvalue, char *, cp, tor_free(cp)); + smartlist_free(*(smartlist_t **)lvalue); + *(smartlist_t **)lvalue = NULL; + } + break; + case CONFIG_TYPE_LINELIST: + case CONFIG_TYPE_LINELIST_S: + config_free_lines(*(config_line_t **)lvalue); + *(config_line_t **)lvalue = NULL; + break; + case CONFIG_TYPE_LINELIST_V: + /* handled by linelist_s. */ + break; + case CONFIG_TYPE_OBSOLETE: + break; + } +} + +/** Clear the option indexed by var in options. Then if + * use_defaults, set it to its default value. + * Called by config_init() and option_reset_line() and option_assign_line(). */ +static void +config_reset(const config_format_t *fmt, void *options, + const config_var_t *var, int use_defaults) +{ + config_line_t *c; + char *msg = NULL; + CONFIG_CHECK(fmt, options); + config_clear(fmt, options, var); /* clear it first */ + if (!use_defaults) + return; /* all done */ + if (var->initvalue) { + c = tor_malloc_zero(sizeof(config_line_t)); + c->key = tor_strdup(var->name); + c->value = tor_strdup(var->initvalue); + if (config_assign_value(fmt, options, c, &msg) < 0) { + log_warn(LD_BUG, "Failed to assign default: %s", msg); + tor_free(msg); /* if this happens it's a bug */ + } + config_free_lines(c); + } +} + +/** Release storage held by options. */ +void +config_free(const config_format_t *fmt, void *options) +{ + int i; + + if (!options) + return; + + tor_assert(fmt); + + for (i=0; fmt->vars[i].name; ++i) + config_clear(fmt, options, &(fmt->vars[i])); + if (fmt->extra) { + config_line_t **linep = STRUCT_VAR_P(options, fmt->extra->var_offset); + config_free_lines(*linep); + *linep = NULL; + } + tor_free(options); +} + +/** Return true iff a and b contain identical keys and values in identical + * order. */ +int +config_lines_eq(config_line_t *a, config_line_t *b) +{ + while (a && b) { + if (strcasecmp(a->key, b->key) || strcmp(a->value, b->value)) + return 0; + a = a->next; + b = b->next; + } + if (a || b) + return 0; + return 1; +} + +/** Return the number of lines in a whose key is key. */ +int +config_count_key(const config_line_t *a, const char *key) +{ + int n = 0; + while (a) { + if (!strcasecmp(a->key, key)) { + ++n; + } + a = a->next; + } + return n; +} + +/** Return true iff the option name has the same value in o1 + * and o2. Must not be called for LINELIST_S or OBSOLETE options. + */ +int +config_is_same(const config_format_t *fmt, + const void *o1, const void *o2, + const char *name) +{ + config_line_t *c1, *c2; + int r = 1; + CONFIG_CHECK(fmt, o1); + CONFIG_CHECK(fmt, o2); + + c1 = config_get_assigned_option(fmt, o1, name, 0); + c2 = config_get_assigned_option(fmt, o2, name, 0); + r = config_lines_eq(c1, c2); + config_free_lines(c1); + config_free_lines(c2); + return r; +} + +/** Copy storage held by old into a new or_options_t and return it. */ +void * +config_dup(const config_format_t *fmt, const void *old) +{ + or_options_t *newopts; + int i; + config_line_t *line; + + newopts = config_new(fmt); + for (i=0; fmt->vars[i].name; ++i) { + if (fmt->vars[i].type == CONFIG_TYPE_LINELIST_S) + continue; + if (fmt->vars[i].type == CONFIG_TYPE_OBSOLETE) + continue; + line = config_get_assigned_option(fmt, old, fmt->vars[i].name, 0); + if (line) { + char *msg = NULL; + if (config_assign(fmt, newopts, line, 0, 0, &msg) < 0) { + log_err(LD_BUG, "config_get_assigned_option() generated " + "something we couldn't config_assign(): %s", msg); + tor_free(msg); + tor_assert(0); + } + } + config_free_lines(line); + } + return newopts; +} +/** Set all vars in the configuration object options to their default + * values. */ +void +config_init(const config_format_t *fmt, void *options) +{ + int i; + const config_var_t *var; + CONFIG_CHECK(fmt, options); + + for (i=0; fmt->vars[i].name; ++i) { + var = &fmt->vars[i]; + if (!var->initvalue) + continue; /* defaults to NULL or 0 */ + config_reset(fmt, options, var, 1); + } +} + +/** Allocate and return a new string holding the written-out values of the vars + * in 'options'. If 'minimal', do not write out any default-valued vars. + * Else, if comment_defaults, write default values as comments. + */ +char * +config_dump(const config_format_t *fmt, const void *default_options, + const void *options, int minimal, + int comment_defaults) +{ + smartlist_t *elements; + const or_options_t *defaults = default_options; + void *defaults_tmp = NULL; + config_line_t *line, *assigned; + char *result; + int i; + char *msg = NULL; + + if (defaults == NULL) { + defaults = defaults_tmp = config_new(fmt); + config_init(fmt, defaults_tmp); + } + + /* XXX use a 1 here so we don't add a new log line while dumping */ + if (default_options == NULL) { + if (fmt->validate_fn(NULL, defaults_tmp, 1, &msg) < 0) { + log_err(LD_BUG, "Failed to validate default config."); + tor_free(msg); + tor_assert(0); + } + } + + elements = smartlist_new(); + for (i=0; fmt->vars[i].name; ++i) { + int comment_option = 0; + if (fmt->vars[i].type == CONFIG_TYPE_OBSOLETE || + fmt->vars[i].type == CONFIG_TYPE_LINELIST_S) + continue; + /* Don't save 'hidden' control variables. */ + if (!strcmpstart(fmt->vars[i].name, "__")) + continue; + if (minimal && config_is_same(fmt, options, defaults, fmt->vars[i].name)) + continue; + else if (comment_defaults && + config_is_same(fmt, options, defaults, fmt->vars[i].name)) + comment_option = 1; + + line = assigned = + config_get_assigned_option(fmt, options, fmt->vars[i].name, 1); + + for (; line; line = line->next) { + smartlist_add_asprintf(elements, "%s%s %s\n", + comment_option ? "# " : "", + line->key, line->value); + } + config_free_lines(assigned); + } + + if (fmt->extra) { + line = *(config_line_t**)STRUCT_VAR_P(options, fmt->extra->var_offset); + for (; line; line = line->next) { + smartlist_add_asprintf(elements, "%s %s\n", line->key, line->value); + } + } + + result = smartlist_join_strings(elements, "", 0, NULL); + SMARTLIST_FOREACH(elements, char *, cp, tor_free(cp)); + smartlist_free(elements); + if (defaults_tmp) + config_free(fmt, defaults_tmp); + return result; +} + +/** Mapping from a unit name to a multiplier for converting that unit into a + * base unit. Used by config_parse_unit. */ +struct unit_table_t { + const char *unit; /**< The name of the unit */ + uint64_t multiplier; /**< How many of the base unit appear in this unit */ +}; + +/** Table to map the names of memory units to the number of bytes they + * contain. */ +static struct unit_table_t memory_units[] = { + { "", 1 }, + { "b", 1<< 0 }, + { "byte", 1<< 0 }, + { "bytes", 1<< 0 }, + { "kb", 1<<10 }, + { "kbyte", 1<<10 }, + { "kbytes", 1<<10 }, + { "kilobyte", 1<<10 }, + { "kilobytes", 1<<10 }, + { "m", 1<<20 }, + { "mb", 1<<20 }, + { "mbyte", 1<<20 }, + { "mbytes", 1<<20 }, + { "megabyte", 1<<20 }, + { "megabytes", 1<<20 }, + { "gb", 1<<30 }, + { "gbyte", 1<<30 }, + { "gbytes", 1<<30 }, + { "gigabyte", 1<<30 }, + { "gigabytes", 1<<30 }, + { "tb", U64_LITERAL(1)<<40 }, + { "terabyte", U64_LITERAL(1)<<40 }, + { "terabytes", U64_LITERAL(1)<<40 }, + { NULL, 0 }, +}; + +/** Table to map the names of time units to the number of seconds they + * contain. */ +static struct unit_table_t time_units[] = { + { "", 1 }, + { "second", 1 }, + { "seconds", 1 }, + { "minute", 60 }, + { "minutes", 60 }, + { "hour", 60*60 }, + { "hours", 60*60 }, + { "day", 24*60*60 }, + { "days", 24*60*60 }, + { "week", 7*24*60*60 }, + { "weeks", 7*24*60*60 }, + { NULL, 0 }, +}; + +/** Table to map the names of time units to the number of milliseconds + * they contain. */ +static struct unit_table_t time_msec_units[] = { + { "", 1 }, + { "msec", 1 }, + { "millisecond", 1 }, + { "milliseconds", 1 }, + { "second", 1000 }, + { "seconds", 1000 }, + { "minute", 60*1000 }, + { "minutes", 60*1000 }, + { "hour", 60*60*1000 }, + { "hours", 60*60*1000 }, + { "day", 24*60*60*1000 }, + { "days", 24*60*60*1000 }, + { "week", 7*24*60*60*1000 }, + { "weeks", 7*24*60*60*1000 }, + { NULL, 0 }, +}; + +/** Parse a string val containing a number, zero or more + * spaces, and an optional unit string. If the unit appears in the + * table u, then multiply the number by the unit multiplier. + * On success, set *ok to 1 and return this product. + * Otherwise, set *ok to 0. + */ +static uint64_t +config_parse_units(const char *val, struct unit_table_t *u, int *ok) +{ + uint64_t v = 0; + double d = 0; + int use_float = 0; + char *cp; + + tor_assert(ok); + + v = tor_parse_uint64(val, 10, 0, UINT64_MAX, ok, &cp); + if (!*ok || (cp && *cp == '.')) { + d = tor_parse_double(val, 0, UINT64_MAX, ok, &cp); + if (!*ok) + goto done; + use_float = 1; + } + + if (!cp) { + *ok = 1; + v = use_float ? DBL_TO_U64(d) : v; + goto done; + } + + cp = (char*) eat_whitespace(cp); + + for ( ;u->unit;++u) { + if (!strcasecmp(u->unit, cp)) { + if (use_float) + v = u->multiplier * d; + else + v *= u->multiplier; + *ok = 1; + goto done; + } + } + log_warn(LD_CONFIG, "Unknown unit '%s'.", cp); + *ok = 0; + done: + + if (*ok) + return v; + else + return 0; +} + +/** Parse a string in the format "number unit", where unit is a unit of + * information (byte, KB, M, etc). On success, set *ok to true + * and return the number of bytes specified. Otherwise, set + * *ok to false and return 0. */ +static uint64_t +config_parse_memunit(const char *s, int *ok) +{ + uint64_t u = config_parse_units(s, memory_units, ok); + return u; +} + +/** Parse a string in the format "number unit", where unit is a unit of + * time in milliseconds. On success, set *ok to true and return + * the number of milliseconds in the provided interval. Otherwise, set + * *ok to 0 and return -1. */ +static int +config_parse_msec_interval(const char *s, int *ok) +{ + uint64_t r; + r = config_parse_units(s, time_msec_units, ok); + if (!ok) + return -1; + if (r > INT_MAX) { + log_warn(LD_CONFIG, "Msec interval '%s' is too long", s); + *ok = 0; + return -1; + } + return (int)r; +} + +/** Parse a string in the format "number unit", where unit is a unit of time. + * On success, set *ok to true and return the number of seconds in + * the provided interval. Otherwise, set *ok to 0 and return -1. + */ +static int +config_parse_interval(const char *s, int *ok) +{ + uint64_t r; + r = config_parse_units(s, time_units, ok); + if (!ok) + return -1; + if (r > INT_MAX) { + log_warn(LD_CONFIG, "Interval '%s' is too long", s); + *ok = 0; + return -1; + } + return (int)r; +} + diff --git a/src/or/confparse.h b/src/or/confparse.h new file mode 100644 index 0000000..f33208e --- /dev/null +++ b/src/or/confparse.h @@ -0,0 +1,132 @@ +/* Copyright (c) 2001 Matej Pfajfar. + * Copyright (c) 2001-2004, Roger Dingledine. + * Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson. + * Copyright (c) 2007-2012, The Tor Project, Inc. */ +/* See LICENSE for licensing information */ + +#ifndef TOR_CONFPARSE_H +#define TOR_CONFPARSE_H + +/** Enumeration of types which option values can take */ +typedef enum config_type_t { + CONFIG_TYPE_STRING = 0, /**< An arbitrary string. */ + CONFIG_TYPE_FILENAME, /**< A filename: some prefixes get expanded. */ + CONFIG_TYPE_UINT, /**< A non-negative integer less than MAX_INT */ + CONFIG_TYPE_INT, /**< Any integer. */ + CONFIG_TYPE_PORT, /**< A port from 1...65535, 0 for "not set", or + * "auto". */ + CONFIG_TYPE_INTERVAL, /**< A number of seconds, with optional units*/ + CONFIG_TYPE_MSEC_INTERVAL,/**< A number of milliseconds, with optional + * units */ + CONFIG_TYPE_MEMUNIT, /**< A number of bytes, with optional units*/ + CONFIG_TYPE_DOUBLE, /**< A floating-point value */ + CONFIG_TYPE_BOOL, /**< A boolean value, expressed as 0 or 1. */ + CONFIG_TYPE_AUTOBOOL, /**< A boolean+auto value, expressed 0 for false, + * 1 for true, and -1 for auto */ + CONFIG_TYPE_ISOTIME, /**< An ISO-formatted time relative to GMT. */ + CONFIG_TYPE_CSV, /**< A list of strings, separated by commas and + * optional whitespace. */ + CONFIG_TYPE_LINELIST, /**< Uninterpreted config lines */ + CONFIG_TYPE_LINELIST_S, /**< Uninterpreted, context-sensitive config lines, + * mixed with other keywords. */ + CONFIG_TYPE_LINELIST_V, /**< Catch-all "virtual" option to summarize + * context-sensitive config lines when fetching. + */ + CONFIG_TYPE_ROUTERSET, /**< A list of router names, addrs, and fps, + * parsed into a routerset_t. */ + CONFIG_TYPE_OBSOLETE, /**< Obsolete (ignored) option. */ +} config_type_t; + +/** An abbreviation for a configuration option allowed on the command line. */ +typedef struct config_abbrev_t { + const char *abbreviated; + const char *full; + int commandline_only; + int warn; +} config_abbrev_t; + +/* Handy macro for declaring "In the config file or on the command line, + * you can abbreviate toks as tok". */ +#define PLURAL(tok) { #tok, #tok "s", 0, 0 } + +/** A variable allowed in the configuration file or on the command line. */ +typedef struct config_var_t { + const char *name; /**< The full keyword (case insensitive). */ + config_type_t type; /**< How to interpret the type and turn it into a + * value. */ + off_t var_offset; /**< Offset of the corresponding member of or_options_t. */ + const char *initvalue; /**< String (or null) describing initial value. */ +} config_var_t; + +/** Represents an English description of a configuration variable; used when + * generating configuration file comments. */ +typedef struct config_var_description_t { + const char *name; + const char *description; +} config_var_description_t; + +/** Type of a callback to validate whether a given configuration is + * well-formed and consistent. See options_trial_assign() for documentation + * of arguments. */ +typedef int (*validate_fn_t)(void*,void*,int,char**); + +/** Information on the keys, value types, key-to-struct-member mappings, + * variable descriptions, validation functions, and abbreviations for a + * configuration or storage format. */ +typedef struct { + size_t size; /**< Size of the struct that everything gets parsed into. */ + uint32_t magic; /**< Required 'magic value' to make sure we have a struct + * of the right type. */ + off_t magic_offset; /**< Offset of the magic value within the struct. */ + config_abbrev_t *abbrevs; /**< List of abbreviations that we expand when + * parsing this format. */ + config_var_t *vars; /**< List of variables we recognize, their default + * values, and where we stick them in the structure. */ + validate_fn_t validate_fn; /**< Function to validate config. */ + /** If present, extra is a LINELIST variable for unrecognized + * lines. Otherwise, unrecognized lines are an error. */ + config_var_t *extra; +} config_format_t; + +/** Macro: assert that cfg has the right magic field for format + * fmt. */ +#define CONFIG_CHECK(fmt, cfg) STMT_BEGIN \ + tor_assert(fmt && cfg); \ + tor_assert((fmt)->magic == \ + *(uint32_t*)STRUCT_VAR_P(cfg,fmt->magic_offset)); \ + STMT_END + +void *config_new(const config_format_t *fmt); +void config_line_append(config_line_t **lst, + const char *key, const char *val); +config_line_t *config_lines_dup(const config_line_t *inp); +void config_free(const config_format_t *fmt, void *options); +int config_lines_eq(config_line_t *a, config_line_t *b); +int config_count_key(const config_line_t *a, const char *key); +config_line_t *config_get_assigned_option(const config_format_t *fmt, + const void *options, const char *key, + int escape_val); +int config_is_same(const config_format_t *fmt, + const void *o1, const void *o2, + const char *name); +void config_init(const config_format_t *fmt, void *options); +void *config_dup(const config_format_t *fmt, const void *old); +char *config_dump(const config_format_t *fmt, const void *default_options, + const void *options, int minimal, + int comment_defaults); +int config_assign(const config_format_t *fmt, void *options, + config_line_t *list, + int use_defaults, int clear_first, char **msg); +config_var_t *config_find_option_mutable(config_format_t *fmt, + const char *key); +const config_var_t *config_find_option(const config_format_t *fmt, + const char *key); + +int config_get_lines(const char *string, config_line_t **result, int extended); +void config_free_lines(config_line_t *front); +const char *config_expand_abbrev(const config_format_t *fmt, + const char *option, + int command_line, int warn_obsolete); + +#endif + diff --git a/src/or/control.c b/src/or/control.c index 4a0f8d7..bb89326 100644 --- a/src/or/control.c +++ b/src/or/control.c @@ -16,6 +16,7 @@ #include "circuitlist.h" #include "circuituse.h" #include "config.h" +#include "confparse.h" #include "connection.h" #include "connection_edge.h" #include "connection_or.h" diff --git a/src/or/dirserv.c b/src/or/dirserv.c index a8f2fb9..9ca4c54 100644 --- a/src/or/dirserv.c +++ b/src/or/dirserv.c @@ -7,6 +7,7 @@ #include "or.h" #include "buffers.h" #include "config.h" +#include "confparse.h" #include "connection.h" #include "connection_or.h" #include "control.h" diff --git a/src/or/include.am b/src/or/include.am index 97072dc..fc41c6d 100644 --- a/src/or/include.am +++ b/src/or/include.am @@ -21,7 +21,8 @@ src_or_libtor_a_SOURCES = \ src/or/circuitlist.c \ src/or/circuituse.c \ src/or/command.c \ - src/or/config.c \ + src/or/config.c \ + src/or/confparse.c \ src/or/connection.c \ src/or/connection_edge.c \ src/or/connection_or.c \ @@ -88,6 +89,7 @@ ORHEADERS = \ src/or/circuituse.h \ src/or/command.h \ src/or/config.h \ + src/or/confparse.h \ src/or/connection.h \ src/or/connection_edge.h \ src/or/connection_or.h \ diff --git a/src/test/test_config.c b/src/test/test_config.c index ff251a2..d9fcd8b 100644 --- a/src/test/test_config.c +++ b/src/test/test_config.c @@ -6,6 +6,7 @@ #include "orconfig.h" #include "or.h" #include "config.h" +#include "confparse.h" #include "connection_edge.h" #include "test.h"

1 0

[tor/master] Merge branch 'confparse_refactor_squashed'
by nickm＠torproject.org 13 Sep '12

13 Sep '12

commit abbde7b1849768c96b3ebf2e08e4f844c0775b9d Merge: 582f218 7d11952 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Sep 13 12:25:45 2012 -0400 Merge branch 'confparse_refactor_squashed' changes/refactor_config | 3 + src/or/circuitbuild.c | 2 + src/or/config.c | 1914 +---------------------------------------------- src/or/config.h | 12 +- src/or/confparse.c | 1226 ++++++++++++++++++++++++++++++ src/or/confparse.h | 132 ++++ src/or/control.c | 1 + src/or/dirserv.c | 1 + src/or/hibernate.c | 1 + src/or/include.am | 6 +- src/or/main.c | 2 + src/or/router.c | 1 + src/or/statefile.c | 606 +++++++++++++++ src/or/statefile.h | 22 + src/or/transports.c | 1 + src/test/test_config.c | 1 + 16 files changed, 2016 insertions(+), 1915 deletions(-)

1 0

[tor/master] mention the bug number in the 6827 changes file
by nickm＠torproject.org 13 Sep '12

13 Sep '12

commit 1e68c213a24237b1733cfee3726aa646a805a5a9 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Sep 13 10:07:06 2012 -0400 mention the bug number in the 6827 changes file --- changes/bug6827 | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/changes/bug6827 b/changes/bug6827 index 799c459..bf71d2b 100644 --- a/changes/bug6827 +++ b/changes/bug6827 @@ -4,5 +4,6 @@ rendezvous/introduction protocols in a hidden service descriptor. Previously, Tor would have confused (as-yet-unused) protocol version numbers greater than 32 with lower ones on many - platforms. Bugfix on 0.2.0.10-alpha; found by George Kadianakis. + platforms. Fixes bug 6827; bugfix on 0.2.0.10-alpha; found by + George Kadianakis.

1 0

[tor/master] Merge remote-tracking branch 'origin/maint-0.2.3'
by nickm＠torproject.org 13 Sep '12

13 Sep '12

commit 582f2187a769ea723f6bf13bc91f7a4b3c861408 Merge: a73dec1 1e68c21 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Sep 13 10:08:51 2012 -0400 Merge remote-tracking branch 'origin/maint-0.2.3' changes/bug6827 | 9 +++++++++ src/or/or.h | 7 +++++-- src/or/routerparse.c | 3 +++ 3 files changed, 17 insertions(+), 2 deletions(-)

1 0

← Newer
1
...
34
35
36
37
38
39
40
...
80
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits September 2012