- tor-commits - lists.torproject.org

r25883: {website} point people at tor-reports because it's more active. (website/trunk/donate/en)
by Andrew Lewman 13 Nov '12

13 Nov '12

Author: phobos Date: 2012-11-13 16:48:51 +0000 (Tue, 13 Nov 2012) New Revision: 25883 Modified: website/trunk/donate/en/thankyou.wml Log: point people at tor-reports because it's more active. Modified: website/trunk/donate/en/thankyou.wml =================================================================== --- website/trunk/donate/en/thankyou.wml 2012-11-13 16:45:50 UTC (rev 25882) +++ website/trunk/donate/en/thankyou.wml 2012-11-13 16:48:51 UTC (rev 25883) @@ -15,8 +15,8 @@ <tr id="thanks_row"> <td id="thanks_l"> <img class="thankyou_h2" src="../images/thankyou_h2.png" height="46" width="195" alt="Keep In Touch!"> - Sign up for our <a href="https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-news">newsletter</a> - <form Method="POST" ACTION="https://lists.torproject.org/cgi-bin/mailman/subscribe/tor-news"> + Sign up for our <a href="https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-reports">newsletter</a> + <form Method="POST" ACTION="https://lists.torproject.org/cgi-bin/mailman/subscribe/tor-reports"> <label class="inactive" for="email">Enter your email address:</label> <input type="text" name="email" id="email" size="25" value="">

1 0

r25882: {website} Adding revised tor project ideas Nick has written four shiny (website/trunk/getinvolved/en)
by Damian Johnson 13 Nov '12

13 Nov '12

Author: atagar Date: 2012-11-13 16:45:50 +0000 (Tue, 13 Nov 2012) New Revision: 25882 Modified: website/trunk/getinvolved/en/volunteer.wml Log: Adding revised tor project ideas Nick has written four shiny, new project ideas (mmmm, shiny...). Modified: website/trunk/getinvolved/en/volunteer.wml =================================================================== --- website/trunk/getinvolved/en/volunteer.wml 2012-11-13 16:22:30 UTC (rev 25881) +++ website/trunk/getinvolved/en/volunteer.wml 2012-11-13 16:45:50 UTC (rev 25882) @@ -358,15 +358,13 @@ extents, but the chief architects are Nick Mathewson and Roger Dingledine. -  <a id="project-jtor"></a> <h3><a href="https://github.com/brl/JTor/wiki">JTor</a> (<a @@ -975,6 +973,126 @@ </li> --> + <a id="limitCapabilities"></a> + <li> + Run With Limited Capabilities + + Priority: Medium to High + + Effort Level: Medium to High + + Skill Level: High + + Likely Mentors: Nick (nickm) + + Many modern operating systems give a running program the ability to drop + capabilities that it no longer needs, and other ways for a program to run + pieces of itself in a sandbox with diminished privileges. + + + + We'd like to do this with Tor, to improve its resistance to attacks. The + easiest areas to address would be on systems like <a + href="https://lwn.net/Articles/475361/">recent Linux kernels</a> that make + it easy to drop or restrict the set of syscalls that a program can invoke. + That's a great project, but probably not big enough for an internship just + on its own. For that, we'd want to make progress on at least multiple + platforms, or look into refactoring Tor into pieces that need more + privileges and pieces that don't with an eye towards sandboxing them + differently. + + + + See tickets <a href="https://trac.torproject.org/7005">#7005</a> and <a + href="https://trac.torproject.org/5219">#5219</a>, and their descendants, + for more information. + + </li> + + <a id="torCleanup"></a> + <li> + Tor Codebase Cleanup + + Priority: Medium to High + + Effort Level: Low to High, depending on subproject chosen + + Skill Level: Medium to High + + Likely Mentors: Nick (nickm) + + The Tor code is almost 10 years old in places, and we haven't always had + enough time or wisdom to write things as well as we could have. Our unit + test coverage is shamefully low, and the dependency graph of our modules is + shamefully convoluted . We could use refactoring and unit tests! Please + look through the Tor source code and look for ugly or tricky code or + dependencies -- the uglier and trickier the better -- and think about how + you could make the code look better, read better, and (subject to testing) + work better. + + + + If this is for a fun side-project, it would be great for you to work on + anything that can be made better and more tested. For an internship-level + position, we'd hope that you could find a number of particularly tricky or + knotty piece of the code to clean up, and aim for resolving the ugliest + problems, not necessarily the easiest. + + + + For a big project here, it would be great to pick one of the major + "submodules" of Tor -- path selection, node discovery, directory authority + operations, directory service -- and refactor its interface completely, to + minify and codify its points of contact with the rest of Tor. + + </li> + + <a id="httpsImersonation"></a> + <li> + HTTPS Server Impersonation + + Priority: Medium + + Effort Level: Medium to High + + Skill Level: Medium to High + + Likely Mentors: Nick (nickm) + + We have an open proposal for a way to make Tor bridges avoid censorship by + impersonating an HTTPS server. Specifically, we need to hack some popular + SSL "reverse proxy" (your choice) so that it relays regular web connections + to an HTTP server, but certain connections to a local Tor process. <a + href="https://gitweb.torproject.org/torspec.git/blob_plain/HEAD:/proposals/203-ht…">Proposal + 203</a> has a general design sketch. + + </li> + + <a id="chutneyExpansion"></a> + <li> + Make Chutney Do More, More Reliably + + Priority: Medium + + Effort Level: Medium to High, depending on scope of project + + Skill Level: Medium + + Likely Mentors: Nick (nickm) + + We have a little Python tool called <a + href="https://gitweb.torproject.org/nickm/chutney.git">Chutney</a> for + making small local test networks. It's small, not widely used, and not as + automated as it could be. + + + + It would be great to see chutney extended and a set of supporting tests + built to the point where we could use Chutney to exercise various Tor + features as an automated integration test. + + </li> + <a id="stemUsability"></a> <li> Stem Usability Improvements

1 0

r25881: {} add scripps presentation from last week. (in projects/presentations: . images)
by Andrew Lewman 13 Nov '12

13 Nov '12

Author: phobos Date: 2012-11-13 16:22:30 +0000 (Tue, 13 Nov 2012) New Revision: 25881 Added: projects/presentations/2012-11-08-Scripps-IIJ-Presentation.pdf projects/presentations/2012-11-08-Scripps-IIJ-Presentation.tex projects/presentations/images/tails-logo-7.png Log: add scripps presentation from last week. Added: projects/presentations/2012-11-08-Scripps-IIJ-Presentation.pdf =================================================================== (Binary files differ) Property changes on: projects/presentations/2012-11-08-Scripps-IIJ-Presentation.pdf ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: projects/presentations/2012-11-08-Scripps-IIJ-Presentation.tex =================================================================== --- projects/presentations/2012-11-08-Scripps-IIJ-Presentation.tex (rev 0) +++ projects/presentations/2012-11-08-Scripps-IIJ-Presentation.tex 2012-11-13 16:22:30 UTC (rev 25881) @@ -0,0 +1,135 @@ +\documentclass{beamer} +\mode<presentation> +\usetheme{Boadilla} +\title{Scripps IIJ} +\author{Andrew Lewman \\ andrew(a)torproject.org} +\date{November 08, 2012} +\begin{document} + +\begin{frame} +\maketitle +\begin{center} +\includegraphics[height=3cm]{./images/2009-tor-logo} +\end{center} +\end{frame} + +\begin{frame} +\frametitle{The Tor Project, Inc.} +501(c)(3) non-profit organization dedicated to the research and development of technologies for online anonymity and privacy +\begin{center} +\includegraphics[height=5cm]{./images/2009-oval_sticker_new} +\end{center} +\end{frame} + + +\begin{frame} +\frametitle{Anonymous communication} +\begin{itemize} +\item People have to hide in a crowd of other people ("anonymity loves company") +\item The goal of the system is to make all users look as similar as possible, to give a bigger crowd +\item Hide who is communicating with whom +\item Layered encryption and random delays hide correlation between input traffic and output traffic +\end{itemize} +\end{frame} + +\begin{frame} +\frametitle{What is Tor?} +\begin{itemize} +\item online anonymity software and network +\pause \item open source, freely available (3-clause BSD license) +\pause \item active research environment: \\ +Drexel, Univ of Waterloo, Georgia Tech, Princeton, Boston University, University College London, Univ of Minnesota, National Science Foundation, Naval Research Labs, Cambridge UK, Bamberg Germany, MIT... +\pause \item increasingly diverse toolset: \\ +Tor, Tor Browser Bundle, Tails LiveCD, Tor Weather, Tor auto-responder, Secure Updater, Orbot, Torora, Tor Check, Arm, Nymble, Tor Control, and so on. +\end{itemize} +\end{frame} + +\begin{frame} +\frametitle{Who uses Tor?} +\parbox{8cm}{\sloppy \setbeamercolor{background}[\includegraphics[scale=0.35]{./images/anonymousman}} +\parbox{3cm}{\sloppy +\begin{flushleft} +\begin{itemize} +\begin{small} +\item Normal people +\item Journalists +\item Law Enforcement +\item Human Rights Activists +\item Business Execs +\item Militaries +\item Abuse Victims +\end{small} +\end{itemize} +\end{flushleft} +} +\end{frame} + +\begin{frame} +\frametitle{Doesn't Tor enable criminals to do bad things?} +\begin{quotation} +\noindent \includegraphics[width=1cm]{./images/opquo}\quad Criminals can already do bad things. Since they're willing to break laws, they already have lots of options available that provide better privacy than Tor provides. +\end{quotation} +\flushright +\small source: https://www.torproject.org/docs/faq-abuse.html.en\#WhatAboutCriminals +\end{frame} + +\begin{frame} +\frametitle{estimated 500k to 900k daily users} +\includegraphics[scale=0.4]{./images/huge-crowd} +\end{frame} + +\begin{frame} +\frametitle{Tor hides communication patterns by relaying data through volunteer servers} +\begin{center} +\begin{overlayarea}{9cm}{6cm} +\only<1>{\includegraphics[width=9cm]{./images/tor-network}} +\only<2>{\includegraphics[width=9cm]{./images/tor-safe-selection}} +\only<3>{\includegraphics[width=9cm]{./images/tor-safe-path}} +\only<4>{\includegraphics[width=9cm]{./images/tor-keys1}} +\end{overlayarea} +\flushright +\tiny Diagram: Robert Watson +\end{center} +\end{frame} + +\begin{frame} +\frametitle{Tor hidden services allow privacy enhanced hosting of services} +\includegraphics[scale=0.5]{./images/hidden-federalist} +\end{frame} + +\begin{frame} +\frametitle{dot onion you say?} +\includegraphics[scale=0.6]{./images/hidden-federalist-zoom} +\end{frame} + +\begin{frame} +\frametitle{Tails Live System/Journalist Toolkit} +\begin{center} +\includegraphics[scale=0.4]{images/tails-logo-7.png} +\end{center} +\begin{itemize} +\item Completely self-contained system +\pause \item Runs from USB drive or CD/DVD completely +\pause \item On-screen keyboard if you can't trust the computer +\pause \item Full office, video editing, audio editing suites +\pause \item Designed to not leak data +\pause \item {\LARGE Learn more at https://tails.boum.org/} +\end{itemize} +\end{frame} + +\begin{frame} +\frametitle{Thanks!} +\includegraphics[scale=0.6]{./images/thankyou_img} +\flushright +Visit \url{https://www.torproject.org/} for more information, links, and ideas. +\end{frame} + +\begin{frame} +\frametitle{Credits \& Thanks} +\begin{itemize} +\item who uses tor? \url{http://www.flickr.com/photos/mattw/2336507468/siz}, Matt Westervelt, CC-BY-SA. +\item 500k, \url{http://www.flickr.com/photos/lukaskracic/334850378/sizes/l/}, Luka Skracic, used with permission. +\end{itemize} +\end{frame} + +\end{document} Property changes on: projects/presentations/2012-11-08-Scripps-IIJ-Presentation.tex ___________________________________________________________________ Added: svn:mime-type + text/x-tex Added: projects/presentations/images/tails-logo-7.png =================================================================== (Binary files differ) Property changes on: projects/presentations/images/tails-logo-7.png ___________________________________________________________________ Added: svn:mime-type + image/png

1 0

r25880: {website} Suggested OPW revisions Couple of revsions suggested by Mari (website/trunk/getinvolved/en)
by Damian Johnson 13 Nov '12

13 Nov '12

Author: atagar Date: 2012-11-13 16:17:13 +0000 (Tue, 13 Nov 2012) New Revision: 25880 Modified: website/trunk/getinvolved/en/volunteer.wml Log: Suggested OPW revisions Couple of revsions suggested by Marina including... * Noting the application deadline date. * Saying that the ideas listing is for OPW in addition to GSoC. Modified: website/trunk/getinvolved/en/volunteer.wml =================================================================== --- website/trunk/getinvolved/en/volunteer.wml 2012-11-13 03:20:32 UTC (rev 25879) +++ website/trunk/getinvolved/en/volunteer.wml 2012-11-13 16:17:13 UTC (rev 25880) @@ -76,9 +76,11 @@ - The program runs from January 2 through April 2, 2013. Like GSoC the - program involves a stipend so, if you're a woman interested in fighting - surveillance and censorship around the world then let us know! + The program runs from January 2 through April 2, 2013. The application + deadline is December 3, 2012 and it is best to start as early as possible + preparing your application. Like GSoC the program involves a stipend so, if + you're a woman interested in fighting surveillance and censorship around + the world then let us know! @@ -767,13 +769,14 @@ <h2><a class="anchor" href="#Coding">Project Ideas</a></h2> - You may find some of these projects to be good <a href="<page - about/gsoc>">Google Summer of Code</a> ideas. We have labelled each idea - with how useful it would be to the overall Tor project (priority), how - much work we expect it would be (effort level), how much clue you should - start with (skill level), and which of our <a href="<page - about/corepeople>">core developers</a> would be good mentors. - If one or more of these ideas looks promising to you, please <a + You may find some of these projects to be good ideas for <a href="<page + about/gsoc>">Google Summer of Code</a> and the <a + href="https://live.gnome.org/OutreachProgramForWomen">Outreach Program for + Women</a>. We have labelled each idea with how useful it would be to the + overall Tor project (priority), how much work we expect it would be (effort + level), how much clue you should start with (skill level), and which of our + <a href="<page about/corepeople>">core developers</a> would be good + mentors. If one or more of these ideas looks promising to you, please <a href="<page about/contact>">contact us</a> to discuss your plans rather than sending blind applications. You may also want to propose your own project idea — which often results in the best applications.

1 0

[webstats/master] Don't delete parsed files from in/.
by runa＠torproject.org 13 Nov '12

13 Nov '12

commit b306f7fc0f0285ea1d008dbbe271495bc6d92b3e Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Nov 7 17:18:43 2012 -0500 Don't delete parsed files from in/. --- src/org/torproject/webstats/Main.java | 63 ++++++++++++-------------------- 1 files changed, 24 insertions(+), 39 deletions(-) diff --git a/src/org/torproject/webstats/Main.java b/src/org/torproject/webstats/Main.java index 1c7780c..7241d68 100644 --- a/src/org/torproject/webstats/Main.java +++ b/src/org/torproject/webstats/Main.java @@ -25,7 +25,7 @@ import org.apache.commons.compress.compressors.gzip.*; * * The main operation is to parse Apache web log files from the in/ * directory and write sanitized web log files to the out/ directory. - * Files in the in/ directory are assumed to never change and will be + * Files in the in/ directory are assumed to never change and may be * deleted after processing by this program. Files in the out/ directory * are guaranteed to never change and may be deleted by a subsequently * running program. @@ -34,8 +34,8 @@ import org.apache.commons.compress.compressors.gzip.*; * in/ are not parsed more than once and that files in out/ do not need to * be changed: * - state/lock prevents concurrent executions of this program. - * - state/in-history contains file names of previously read and deleted - * files in the in/ directory. + * - state/in-history contains file names of previously read and possibly + * deleted files in the in/ directory. * - state/in-history.new is the file written in the current execution * that will replace state/in-history during the execution. * - state/temp/ contains new or updated output files parsed in the @@ -56,12 +56,12 @@ import org.apache.commons.compress.compressors.gzip.*; * 2. Read the contents from state/in-history and state/out-history and * the directory listings of in/ to memory. * 3. For each file in in/: - * a. Append the file name to state/in-history.new. - * b. Check that the file name is not contained in state/in-history. - * If it is, print out a warning and skip the file. - * c. Parse and sanitize the file in chunks of 250,000 lines to reduce + * a. Append the file name to state/in-history.new if it was not + * contained in state/in-history. If it was contained, skip the + * file. + * b. Parse and sanitize the file in chunks of 250,000 lines to reduce * writes. - * d. When writing sanitized chunks to output files, for each output + * c. When writing sanitized chunks to output files, for each output * file, check in the following order if there is already such a * file in * i. state/temp/, @@ -73,8 +73,7 @@ import org.apache.commons.compress.compressors.gzip.*; * 4. Rename state/in-history to state/in-history.old and rename * state/in-history.new to state/in-history. Delete * state/in-history.old. - * 5. Delete files in in/ that have been parsed in this execution. - * 6. For each file in state/temp/: + * 5. For each file in state/temp/: * a. Check if there's a corresponding line in state/out-history. If * so, check whether there is a file in state/full/ or out/. If * so, move the file to state/full/. Otherwise move the file to @@ -84,13 +83,13 @@ import org.apache.commons.compress.compressors.gzip.*; * days old, move the file to state/full/. * c. If b. does not apply, append a line to out-history.new and move * the file to out/. - * 7. For each file in state/full/, check whether the sanitized log is at + * 6. For each file in state/full/, check whether the sanitized log is at * least four (4) days old and not contained in state/out-history. If * so, append a line to out-history.new and move the file to out/. - * 8. Rename state/out-history to state/out-history.old and rename + * 7. Rename state/out-history to state/out-history.old and rename * state/out-history.new to state/out-history. Delete * state/out-history.old. - * 9. Delete state/lock and exit. + * 8. Delete state/lock and exit. * * If the program is interrupted and leaves a lock file in state/lock, it * requires an operator to fix the state/ directory and make it work @@ -133,20 +132,19 @@ public class Main { readHistoryFiles(); /* Step 2 */ readInDirectoryListing(); for (File inFile : inFiles) { /* Step 3 */ - appendToInHistory(inFile); + appendToInHistoryIfNotContained(inFile); if (!checkFileName(inFile) || checkParsedBefore(inFile)) { continue; } sanitizeInFile(inFile); } overwriteInHistoryFile(); /* Step 4 */ - deleteParsedInFiles(); /* Step 5 */ - for (String outputFileName : updatedOutputFiles) { /* Step 6 */ + for (String outputFileName : updatedOutputFiles) { /* Step 5 */ moveOutputFile(outputFileName); } - moveFullFilesToOut(); /* Step 7 */ - overwriteOutHistoryFile(); /* Step 8 */ - deleteLockFile(); /* Step 9 */ + moveFullFilesToOut(); /* Step 6 */ + overwriteOutHistoryFile(); /* Step 7 */ + deleteLockFile(); /* Step 8 */ } /* Define file and directory names. */ @@ -262,10 +260,12 @@ public class Main { } return result; } - private static void appendToInHistory(File inFile) { - inHistoryNewFiles.add(inFile.getAbsolutePath()); - String line = inFile.getAbsolutePath(); - appendToHistoryFile(stateInHistoryNewFile, line); + private static void appendToInHistoryIfNotContained(File inFile) { + if (!inHistoryNewFiles.contains(inFile.getAbsolutePath())) { + inHistoryNewFiles.add(inFile.getAbsolutePath()); + String line = inFile.getAbsolutePath(); + appendToHistoryFile(stateInHistoryNewFile, line); + } } private static void appendToHistoryFile(File historyFile, String line) { try { @@ -286,14 +286,7 @@ public class Main { inFile.getName().endsWith(".gz"); } private static boolean checkParsedBefore(File inFile) { - if (inHistoryFiles.contains(inFile.getAbsolutePath())) { - System.err.println("Parsed and subsequently deleted input file '" - + inFile.getAbsolutePath() + "' before. It shouldn't be " - + "there again. Skipping it now and not deleting it later."); - return true; - } else { - return false; - } + return inHistoryFiles.contains(inFile.getAbsolutePath()); } private static void sanitizeInFile(File inFile) { try { @@ -452,14 +445,6 @@ public class Main { stateInHistoryNewFile.renameTo(stateInHistoryFile); stateInHistoryOldFile.delete(); } - private static void deleteParsedInFiles() { - Set<String> filesToDelete = new HashSet<String>(); - filesToDelete.addAll(inHistoryNewFiles); - filesToDelete.removeAll(inHistoryFiles); - for (String file : filesToDelete) { - new File(file).delete(); - } - } private static void moveOutputFile(String outputFileName) { File outFile = new File(outDirectory, outputFileName); File stateTempFile = new File(stateTempDirectory, outputFileName);

1 0

[webstats/master] Rearrange steps to make it easier to recover from failures.
by runa＠torproject.org 13 Nov '12

13 Nov '12

commit 287216d2b141828f6cf108c91f94733a5d2fc7f2 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Nov 7 19:39:12 2012 -0500 Rearrange steps to make it easier to recover from failures. --- src/org/torproject/webstats/Main.java | 55 ++++++++++++++++++++------------ 1 files changed, 34 insertions(+), 21 deletions(-) diff --git a/src/org/torproject/webstats/Main.java b/src/org/torproject/webstats/Main.java index 4bb035f..378093e 100644 --- a/src/org/torproject/webstats/Main.java +++ b/src/org/torproject/webstats/Main.java @@ -50,11 +50,13 @@ import org.apache.commons.compress.compressors.gzip.*; * - state/diff/ contains new parts for files in the out/ directory which * have been deleted. * - * The steps taken by this program are as follows: + * The phases and steps taken by this program are as follows: + * + * Phase I: Read files from in/, sanitize them, and write them to state/. * 1. Check that state/lock does not exists, or exit immediately. Add a * new state/lock file. - * 2. Read the contents from state/in-history and state/out-history and - * the directory listings of in/ to memory. + * 2. Read the contents from state/in-history and the directory listing + * of in/ to memory. * 3. For each file in in/: * a. Append the file name to state/in-history.new if it was not * contained in state/in-history. If it was contained, skip the @@ -73,7 +75,10 @@ import org.apache.commons.compress.compressors.gzip.*; * 4. Rename state/in-history to state/in-history.old and rename * state/in-history.new to state/in-history. Delete * state/in-history.old. - * 5. For each file in state/temp/: + * + * Phase II: Move files that won't change anymore from state/ to out/. + * 5. Read the contents from state/out-history to memory. + * 6. For each file in state/temp/: * a. Check if there's a corresponding line in state/out-history. If * so, check whether there is a file in state/full/ or out/. If * so, move the file to state/full/. Otherwise move the file to @@ -83,13 +88,13 @@ import org.apache.commons.compress.compressors.gzip.*; * days old, move the file to state/full/. * c. If b. does not apply, append a line to out-history.new and move * the file to out/. - * 6. For each file in state/full/, check whether the sanitized log is at + * 7. For each file in state/full/, check whether the sanitized log is at * least four (4) days old and not contained in state/out-history. If * so, append a line to out-history.new and move the file to out/. - * 7. Rename state/out-history to state/out-history.old and rename + * 8. Rename state/out-history to state/out-history.old and rename * state/out-history.new to state/out-history. Delete * state/out-history.old. - * 8. Delete state/lock and exit. + * 9. Delete state/lock and exit. * * If the program is interrupted and leaves a lock file in state/lock, it * requires an operator to fix the state/ directory and make it work @@ -98,16 +103,17 @@ import org.apache.commons.compress.compressors.gzip.*; * happen. It may make sense to try a solution in a test environment * first: * A. The file state/in-history.new does not exist and there are no files - * in state/temp/. The process died before step 3. Delete state/lock - * and re-run the program. + * in state/temp/. The process died before step 3, that is before + * actually doing anything of phase I. Delete state/lock and re-run + * the program. * B. The file state/in-history.new exists and there are files in - * state/temp/. The process died during steps 3 or 4. Delete all - * files in state/temp/. If state/in-history does not exist but - * state/in-history.old does exist, rename the latter to the former. - * Delete state/lock and re-run the program. + * state/temp/. The process died during steps 3 or 4, that is, during + * phase I. Delete all files in state/temp/. If state/in-history + * does not exist but state/in-history.old does exist, rename the + * latter to the former. Delete state/lock and re-run the program. * C. The file state/in-history.new does not exist, but there are files - * in state/temp/. The process died after step 4. Run the steps 5 to - * 8 manually. Then re-run the program. + * in state/temp/. The process died after step 4, that is during + * phase II. Run the steps 5 to 9 manually. Then re-run the program. * * Whenever logs are parsed that are 4 days old or older, there may * already be output files in out/ that cannot be modified anymore. The @@ -128,8 +134,10 @@ public class Main { /* Run the steps described above. */ public static void main(String[] args) { + + /* Phase I */ checkAndCreateLockFile(); /* Step 1 */ - readHistoryFiles(); /* Step 2 */ + readInHistoryFile(); /* Step 2 */ readInDirectoryListing(); for (File inFile : inFiles) { /* Step 3 */ appendToInHistoryIfNotContained(inFile); @@ -139,12 +147,15 @@ public class Main { sanitizeInFile(inFile); } overwriteInHistoryFile(); /* Step 4 */ - for (String outputFileName : updatedOutputFiles) { /* Step 5 */ + + /* Phase II */ + readOutHistoryFile(); /* Step 5 */ + for (String outputFileName : updatedOutputFiles) { /* Step 6 */ moveOutputFile(outputFileName); } - moveFullFilesToOut(); /* Step 6 */ - overwriteOutHistoryFile(); /* Step 7 */ - deleteLockFile(); /* Step 8 */ + moveFullFilesToOut(); /* Step 7 */ + overwriteOutHistoryFile(); /* Step 8 */ + deleteLockFile(); /* Step 9 */ } /* Define file and directory names. */ @@ -209,10 +220,12 @@ public class Main { System.exit(1); } } - private static void readHistoryFiles() { + private static void readInHistoryFile() { inHistoryFiles = readAndCopyHistoryFile(stateInHistoryFile, stateInHistoryNewFile); inHistoryNewFiles = new HashSet<String>(inHistoryFiles); + } + private static void readOutHistoryFile() { outHistoryFiles = readAndCopyHistoryFile(stateOutHistoryFile, stateOutHistoryNewFile); }

1 0

[webstats/master] Fix typo in documentation.
by runa＠torproject.org 13 Nov '12

13 Nov '12

commit 1b197f732a4e8079546ea9107fe76f27d4d0d542 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Nov 7 16:50:23 2012 -0500 Fix typo in documentation. --- src/org/torproject/webstats/Main.java | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/org/torproject/webstats/Main.java b/src/org/torproject/webstats/Main.java index 92732cd..1c7780c 100644 --- a/src/org/torproject/webstats/Main.java +++ b/src/org/torproject/webstats/Main.java @@ -21,7 +21,7 @@ import org.apache.commons.compress.compressors.gzip.*; * 8. Override time with '00:00:00 +0000'. * 9. Override referer (sic!) with '"-"'. * 10. Override user agent with '"-"'. - * 11. Truncate resouce at the first '?' character. + * 11. Truncate resource at the first '?' character. * * The main operation is to parse Apache web log files from the in/ * directory and write sanitized web log files to the out/ directory.

1 0

[webstats/master] .gz-compress newly written output files to save disk space.
by runa＠torproject.org 13 Nov '12

13 Nov '12

commit 7bb3197fca8683c0a1f2d13cfacefda7f5f17d92 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Nov 7 19:02:57 2012 -0500 .gz-compress newly written output files to save disk space. --- src/org/torproject/webstats/Main.java | 55 ++++++++++++++++++++++++++++----- 1 files changed, 47 insertions(+), 8 deletions(-) diff --git a/src/org/torproject/webstats/Main.java b/src/org/torproject/webstats/Main.java index 7241d68..4bb035f 100644 --- a/src/org/torproject/webstats/Main.java +++ b/src/org/torproject/webstats/Main.java @@ -328,36 +328,74 @@ public class Main { } private static void storeOutputFile(String outputFileName, List<String> cachedLinesList) { - updatedOutputFiles.add(outputFileName); + String outputGzFileName = outputFileName + ".gz"; File stateTempFile = new File(stateTempDirectory, outputFileName); + File stateTempGzFile = new File(stateTempDirectory, outputGzFileName); File stateFullFile = new File(stateFullDirectory, outputFileName); + File stateFullGzFile = new File(stateFullDirectory, outputGzFileName); File outFile = new File(outDirectory, outputFileName); + File outGzFile = new File(outDirectory, outputGzFileName); File stateDiffFile = new File(stateDiffDirectory, outputFileName); + File stateDiffGzFile = new File(stateDiffDirectory, outputGzFileName); if (stateTempFile.exists()) { + updatedOutputFiles.add(outputFileName); File stateTempOldFile = new File(stateTempDirectory, outputFileName + ".old"); stateTempFile.renameTo(stateTempOldFile); mergeOutputFile(stateTempOldFile, cachedLinesList, stateTempFile); stateTempOldFile.delete(); + } else if (stateTempGzFile.exists()) { + updatedOutputFiles.add(outputGzFileName); + File stateTempGzOldFile = new File(stateTempDirectory, + outputGzFileName + ".old"); + stateTempGzFile.renameTo(stateTempGzOldFile); + mergeOutputFile(stateTempGzOldFile, cachedLinesList, + stateTempGzFile); + stateTempGzOldFile.delete(); } else if (stateFullFile.exists()) { + updatedOutputFiles.add(outputFileName); mergeOutputFile(stateFullFile, cachedLinesList, stateTempFile); + } else if (stateFullGzFile.exists()) { + updatedOutputFiles.add(outputGzFileName); + mergeOutputFile(stateFullGzFile, cachedLinesList, stateTempGzFile); } else if (outFile.exists()) { + updatedOutputFiles.add(outputFileName); mergeOutputFile(outFile, cachedLinesList, stateTempFile); + } else if (outGzFile.exists()) { + updatedOutputFiles.add(outputGzFileName); + mergeOutputFile(outGzFile, cachedLinesList, stateTempGzFile); } else if (stateDiffFile.exists()) { + updatedOutputFiles.add(outputFileName); mergeOutputFile(stateDiffFile, cachedLinesList, stateTempFile); + } else if (stateDiffGzFile.exists()) { + updatedOutputFiles.add(outputGzFileName); + mergeOutputFile(stateDiffGzFile, cachedLinesList, stateTempGzFile); } else { - writeNewOutputFile(cachedLinesList, stateTempFile); + updatedOutputFiles.add(outputGzFileName); + writeNewOutputFile(cachedLinesList, stateTempGzFile); } } private static void mergeOutputFile(File oldOutputFile, List<String> cachedLinesList, File newOutputFile) { try { - BufferedReader br = new BufferedReader( - new FileReader(oldOutputFile)); + BufferedReader br; + if (oldOutputFile.getName().endsWith(".gz")) { + br = new BufferedReader(new InputStreamReader( + new GzipCompressorInputStream(new FileInputStream( + oldOutputFile)))); + } else { + br = new BufferedReader(new FileReader(oldOutputFile)); + } String line; newOutputFile.getParentFile().mkdirs(); - BufferedWriter bw = new BufferedWriter( - new FileWriter(newOutputFile)); + BufferedWriter bw; + if (newOutputFile.getName().endsWith(".gz")) { + bw = new BufferedWriter(new OutputStreamWriter( + new GzipCompressorOutputStream(new FileOutputStream( + newOutputFile)))); + } else { + bw = new BufferedWriter(new FileWriter(newOutputFile)); + } int cachedLinesListPosition = 0, totalCachedLines = cachedLinesList.size(); while ((line = br.readLine()) != null) { @@ -388,8 +426,9 @@ public class Main { File outputFile) { try { outputFile.getParentFile().mkdirs(); - BufferedWriter bw = new BufferedWriter( - new FileWriter(outputFile)); + BufferedWriter bw = new BufferedWriter(new OutputStreamWriter( + new GzipCompressorOutputStream(new FileOutputStream( + outputFile)))); for (String cachedLine : cachedLinesList) { bw.write(cachedLine + "\n"); }

1 0

[tor/master] bump to 0.2.4.6-alpha-dev
by arma＠torproject.org 13 Nov '12

13 Nov '12

commit 83b1a50cc0ba7e349500de13e5c22ad4db83df39 Author: Roger Dingledine <arma(a)torproject.org> Date: Tue Nov 13 01:46:59 2012 -0500 bump to 0.2.4.6-alpha-dev --- configure.ac | 2 +- contrib/tor-mingw.nsi.in | 2 +- src/win32/orconfig.h | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 2a2b8a4..9963a97 100644 --- a/configure.ac +++ b/configure.ac @@ -3,7 +3,7 @@ dnl Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson dnl Copyright (c) 2007-2012, The Tor Project, Inc. dnl See LICENSE for licensing information -AC_INIT([tor],[0.2.4.6-alpha]) +AC_INIT([tor],[0.2.4.6-alpha-dev]) AC_CONFIG_SRCDIR([src/or/main.c]) AM_INIT_AUTOMAKE m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) diff --git a/contrib/tor-mingw.nsi.in b/contrib/tor-mingw.nsi.in index ae79fe3..c22d0c5 100644 --- a/contrib/tor-mingw.nsi.in +++ b/contrib/tor-mingw.nsi.in @@ -8,7 +8,7 @@ !include "LogicLib.nsh" !include "FileFunc.nsh" !insertmacro GetParameters -!define VERSION "0.2.4.6-alpha" +!define VERSION "0.2.4.6-alpha-dev" !define INSTALLER "tor-${VERSION}-win32.exe" !define WEBSITE "https://www.torproject.org/" !define LICENSE "LICENSE" diff --git a/src/win32/orconfig.h b/src/win32/orconfig.h index a08cde2..8fe31a0 100644 --- a/src/win32/orconfig.h +++ b/src/win32/orconfig.h @@ -232,7 +232,7 @@ #define USING_TWOS_COMPLEMENT /* Version number of package */ -#define VERSION "0.2.4.6-alpha" +#define VERSION "0.2.4.6-alpha-dev"

1 0

[tor/master] bump to 0.2.4.6-alpha
by arma＠torproject.org 13 Nov '12

13 Nov '12

commit b13c6becc892d9716d8645f16b5d03bdb0b34651 Author: Roger Dingledine <arma(a)torproject.org> Date: Tue Nov 13 00:00:33 2012 -0500 bump to 0.2.4.6-alpha --- configure.ac | 2 +- contrib/tor-mingw.nsi.in | 2 +- src/win32/orconfig.h | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 105528e..2a2b8a4 100644 --- a/configure.ac +++ b/configure.ac @@ -3,7 +3,7 @@ dnl Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson dnl Copyright (c) 2007-2012, The Tor Project, Inc. dnl See LICENSE for licensing information -AC_INIT([tor],[0.2.4.5-alpha-dev]) +AC_INIT([tor],[0.2.4.6-alpha]) AC_CONFIG_SRCDIR([src/or/main.c]) AM_INIT_AUTOMAKE m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) diff --git a/contrib/tor-mingw.nsi.in b/contrib/tor-mingw.nsi.in index 427fa43..ae79fe3 100644 --- a/contrib/tor-mingw.nsi.in +++ b/contrib/tor-mingw.nsi.in @@ -8,7 +8,7 @@ !include "LogicLib.nsh" !include "FileFunc.nsh" !insertmacro GetParameters -!define VERSION "0.2.4.5-alpha-dev" +!define VERSION "0.2.4.6-alpha" !define INSTALLER "tor-${VERSION}-win32.exe" !define WEBSITE "https://www.torproject.org/" !define LICENSE "LICENSE" diff --git a/src/win32/orconfig.h b/src/win32/orconfig.h index e4b2df7..a08cde2 100644 --- a/src/win32/orconfig.h +++ b/src/win32/orconfig.h @@ -232,7 +232,7 @@ #define USING_TWOS_COMPLEMENT /* Version number of package */ -#define VERSION "0.2.4.5-alpha-dev" +#define VERSION "0.2.4.6-alpha"

1 0