August 2018 - tor-commits - lists.torproject.org

[policies/master] Add company policy douments and job descriptions.
by linus＠torproject.org 15 Aug '18

15 Aug '18

commit 094f4e844bcfc9e01fb05ef230de9c8cdb926abe Author: Linus Nordberg <linus(a)nordberg.se> Date: Wed Aug 15 13:28:46 2018 +0200 Add company policy douments and job descriptions. Also, add a top level README file. --- README.md | 6 ++ jobdescs/index.html | 23 +++++ jobdescs/jobdesc_ED.txt | 93 +++++++++++++++++++++ jobdescs/jobdesc_P.txt | 65 ++++++++++++++ jobdescs/jobdesc_VP.txt | 51 +++++++++++ policies/TPI_Employee_Communications_Policy.pdf | Bin 0 -> 78339 bytes policies/TPI_Harassment_Prevention_Policy.pdf | Bin 0 -> 98932 bytes policies/TPI_Internal_Complaint_Review_Process.pdf | Bin 0 -> 78416 bytes policies/TPI_Relationships_Policy.pdf | Bin 0 -> 70495 bytes policies/index.html | 34 ++++++++ 10 files changed, 272 insertions(+) diff --git a/README.md b/README.md new file mode 100644 index 0000000..bff607d --- /dev/null +++ b/README.md @@ -0,0 +1,6 @@ +This repository contains the following types of +Tor Project Inc. documents: + +* [Founding documents](corpdocs) +* [Policy documents](policies) +* [Job descriptions](jobdescs) diff --git a/jobdescs/index.html b/jobdescs/index.html new file mode 100644 index 0000000..a68c5f8 --- /dev/null +++ b/jobdescs/index.html @@ -0,0 +1,23 @@ +<html><head><title>Tor Project, Inc. Policy Documents</title></head> + <body> + + <ul> + <li><a href="jobdesc_ED.txt">Internet Freedom Nonprofit Seeks Executive Director</a> + <br> + </li> + </ul> + + <ul> + <li><a href="jobdesc_P.txt">Job Description, President</a> + <br> + </li> + </ul> + + <ul> + <li><a href="jobdesc_VP.txt">Job Description, Vice President</a> + <br> + </li> + </ul> + + </body> +</html> diff --git a/jobdescs/jobdesc_ED.txt b/jobdescs/jobdesc_ED.txt new file mode 100644 index 0000000..d6ca80f --- /dev/null +++ b/jobdescs/jobdesc_ED.txt @@ -0,0 +1,93 @@ +Internet Freedom Nonprofit Seeks Executive Director + +The Tor Project, Inc., is a 501(c)(3) organization headquartered in +Seattle that provides the technical infrastructure for privacy +protection over the Internet, helping millions of activists, +journalists and others around the world communicate securely. With +paid staff and contractors of around 35 engineers and operational +support people, plus many volunteers all over the world who contribute +to our work, the Tor Project is funded in part by government grants +and contracts, as well as by individual, foundation, and corporate +donations. Our mission is "To advance human rights and freedoms by +creating and deploying free and open anonymity and privacy +technologies, supporting their unrestricted availability and use, and +furthering their scientific and popular understanding." + +We are currently seeking an Executive Director to lead the +organization into our next phase of growth and sustainability. This +position can be based anywhere, as our work is international in +scope. This person must be able to communicate regularly with the +staff working on the ground at our organizational headquarters in +Seattle. This position reports directly to the Tor Project Board of +Directors. + +The Executive Director sets the tone for the organization, thinks +strategically, and provides leadership and guidance in the day-to-day +operations. The Tor Project does open source software development, and +the Executive Director must be able to engage with various different +stakeholders in managing the organization. Some of the key attributes +of this person should be: + +* A strong public commitment to privacy, and a good understanding of + related debates. + +* A good understanding of open source development culture, and a + commitment to transparent organizations. + +* A strong commitment to creating and maintaining a healthy and safe + working environment, including for remote participants. + +* A talent for consensus and community building. + +* An ability to effectively support and collaborate with diverse + groups of people. + +* A willingness to share responsibility and leadership of the + organization with a leadership team. + +* An enthusiasm for coaching and developing emerging leaders. + +* A natural leadership ability that encourages and empowers others to + do their best work. + +* Strong written and verbal communications skills. + +Some of the key requirements of this position are at least five years +or equivalent experience in each of the following areas: + +* Nonprofit fundraising, including raising money from government + contracts and grants, foundations, corporations and individual + contributions; + +* Talking to and working with the press, including being active and + visible in promoting organizational goals and enhancing the + organization's image; + +* Solid, hands-on, budget management skills, including budget + preparation, analysis, decision-making and reporting; + +* Human resource administration, including working with a leadership + team and hiring and firing experience; + +* Working closely with a board of directors; + +* Non-profit operations, including managing an office and planning + events, ideally remotely; and + +* Running a corporation of at least $3 million in yearly revenue, + including an ability to recognize when outside professional + assistance is required. + +The successful candidate will have a global perspective on privacy and +censorship issues and an ability to balance "survival issues" (like +next year's funding) with "mission issues" (like making sure we're +actually moving our programs in a good direction). This person will +have the ability to work remotely and work with remote workers. To +this end, this person must be willing to adapt to daily use of privacy +enhancing technologies used to communicate with others. Prior +experience working with software development or internet people is a +plus. + +The Tor Project has made a commitment to being a safe, fair, and +diverse workplace. Our new Executive Director must be committed to +these values and to continuing to foster a welcoming work environment. diff --git a/jobdescs/jobdesc_P.txt b/jobdescs/jobdesc_P.txt new file mode 100644 index 0000000..98ed7f0 --- /dev/null +++ b/jobdescs/jobdesc_P.txt @@ -0,0 +1,65 @@ +Job Description + +President, The Tor Project, Inc. + +The President of the Tor Project, Inc., is a title of authority that +has grown around the role of Tor Project cofounder Roger +Dingledine. The President is an officer of the corporation, is a +member of the staff, reports to the Executive Director, and works +collaboratively with the Board of Directors. The President is a key +leader of the organization and advises the Executive Director on +fiscal, operational, fundraising, marketing, human resource, +technology, and programmatic strategies, as requested. The President +is a signatory on checks and provides secondary review of expenditures +over $5,000. + +Responsibilities + +Strategic vision and leadership + +* Collaborate with the Executive Director and the Board to ensure that + the budget, staff and priorities are aligned with the Tor Project, + Inc.'s, core mission. + +* Provide inspirational leadership and direction to the Tor Community + while ensuring the continued development and management of a + professional and efficient organization. + +* Cultivate a strong and transparent working relationship with the Tor + Project Board, and ensure open communication about the measurement + of financial, programmatic, and impact performance against stated + milestones and goals. + +* In partnership with the Executive Director, help build a diverse and + inclusive Board representative of the Community that is highly + engaged and willing to leverage and secure resources. + +* Provide historical perspective and share lessons learned with the + Executive Director and Board. + +Development + +* Help with fundraising, particularly with funders with whom the + President has personal relationships. + +* Help formulate and execute comprehensive marketing, branding and + development strategies that will ensure consistency throughout the + organization and enhance revenue from major donors, foundations, + government agencies, and corporations + + +Strengthening infrastructure and operations + +* Ensure the delivery of high quality services while managing for + current and future growth. + +* Support and motivate the organization's Staff and Community. + +* Facilitate cross-departmental collaboration and strengthen internal + communications with staff throughout the organization; create and + promote a positive, multicultural work environment that supports + consistency throughout the organization. + +* Call special meetings of the Board as appropriate and guided by the + organization's bylaws. Perform such other duties as may be + requested by the Board. diff --git a/jobdescs/jobdesc_VP.txt b/jobdescs/jobdesc_VP.txt new file mode 100644 index 0000000..1c3fea6 --- /dev/null +++ b/jobdescs/jobdesc_VP.txt @@ -0,0 +1,51 @@ +Job Description + +Vice President, The Tor Project, Inc. + +The Vice President of the Tor Project, Inc., is a title of authority +that has grown around the role of Tor Project cofounder Nick +Mathewson. The Vice President is an officer of the corporation, is a +member of the staff, reports to the Executive Director, and works +collaboratively with the Board of Directors. The Vice President is a +key leader of the organization and advises the Executive Director on +fiscal, operational, fundraising, marketing, human resource, +technology, and programmatic strategies, as requested. The Vice +President is a signatory on checks and provides secondary review of +expenditures over $5,000. + +Responsibilities + +Strategic vision and leadership + +* Collaborate with the Executive Director and the Board to ensure that + the budget, staff and priorities are aligned with the Tor Project, + Inc.'s, core mission. + +* Provide inspirational leadership and direction to the Tor Community + while ensuring the continued development and management of a + professional and efficient organization. + +* Cultivate a strong and transparent working relationship with the Tor + Project Board, and ensure open communication about the measurement + of financial, programmatic, and impact performance against stated + milestones and goals. + + +* In partnership with the Executive Director, help build a diverse and + inclusive Board representative of the Community that is highly + engaged and willing to leverage and secure resources. + +* Provide historical perspective and share lessons learned with the + Executive Director and Board. + +Strengthening infrastructure and operations + +* Ensure the delivery of high quality services while managing for + current and future growth. + +* Support and motivate the organization's Staff and Community. + +* Facilitate cross-departmental collaboration and strengthen internal + communications with staff throughout the organization; create and + promote a positive, multicultural work environment that supports + consistency throughout the organization. diff --git a/policies/TPI_Employee_Communications_Policy.pdf b/policies/TPI_Employee_Communications_Policy.pdf new file mode 100644 index 0000000..d557650 Binary files /dev/null and b/policies/TPI_Employee_Communications_Policy.pdf differ diff --git a/policies/TPI_Harassment_Prevention_Policy.pdf b/policies/TPI_Harassment_Prevention_Policy.pdf new file mode 100644 index 0000000..3873f9c Binary files /dev/null and b/policies/TPI_Harassment_Prevention_Policy.pdf differ diff --git a/policies/TPI_Internal_Complaint_Review_Process.pdf b/policies/TPI_Internal_Complaint_Review_Process.pdf new file mode 100644 index 0000000..43d277d Binary files /dev/null and b/policies/TPI_Internal_Complaint_Review_Process.pdf differ diff --git a/policies/TPI_Relationships_Policy.pdf b/policies/TPI_Relationships_Policy.pdf new file mode 100644 index 0000000..7b3a38b Binary files /dev/null and b/policies/TPI_Relationships_Policy.pdf differ diff --git a/policies/index.html b/policies/index.html new file mode 100644 index 0000000..28a597f --- /dev/null +++ b/policies/index.html @@ -0,0 +1,34 @@ +<html><head><title>Tor Project, Inc. Policy Documents</title></head> + <body> + + <ul> + <li><a href="TPI_Employee_Communications_Policy.pdf">Employee Communications Policy</a> + <br> + </li> + </ul> + + <ul> + <li><a href="TPI_Harassment_Prevention_Policy.pdf">Policy + prohibiting discrimination, harassment, and retaliation</a> + <br> + </li> + </ul> + + <ul> + <li><a href="TPI_Internal_Complaint_Review_Process.pdf">Internal Complaint Review Process</a> + <br> + </li> + </ul> + + <ul> + <li><a href="TPI_Relationships_Policy.pdf">Conflicts of interest, including relationships</a> + <br> + </li> + </ul> + + </body> +</html> + + + +

1 0

[translation/abouttor-homepage] Update translations for abouttor-homepage
by translation＠torproject.org 15 Aug '18

15 Aug '18

commit d615bcbba7b2df1b3bb17bb9d278861ed70ed9fd Author: Translation commit bot <translation(a)torproject.org> Date: Wed Aug 15 11:15:04 2018 +0000 Update translations for abouttor-homepage --- da/aboutTor.dtd | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/da/aboutTor.dtd b/da/aboutTor.dtd index 74748b623..9bd057c86 100644 --- a/da/aboutTor.dtd +++ b/da/aboutTor.dtd @@ -6,20 +6,20 @@ <!ENTITY aboutTor.title "Om Tor"> -<!ENTITY aboutTor.ready.label "Explore. Privately."> -<!ENTITY aboutTor.ready2.label "You’re ready for the world’s most private browsing experience."> +<!ENTITY aboutTor.ready.label "Udforsk. Privat."> +<!ENTITY aboutTor.ready2.label "Du er klar til verdenens mest private browseroplevelse."> <!ENTITY aboutTor.failure.label "Noget gik galt!"> <!ENTITY aboutTor.failure2.label "Tor virker ikke i denne browser."> -<!ENTITY aboutTor.search.label "Search with DuckDuckGo"> +<!ENTITY aboutTor.search.label "Søg med DuckDuckGo"> <!ENTITY aboutTor.searchDDGPost.link "https://duckduckgo.com"> <!ENTITY aboutTor.torbrowser_user_manual_questions.label "Spørgsmål?"> -<!ENTITY aboutTor.torbrowser_user_manual_link.label "Check our Tor Browser Manual »"> +<!ENTITY aboutTor.torbrowser_user_manual_link.label "Tjek vores Tor Browser-manual »">  <!ENTITY aboutTor.torbrowser_user_manual.accesskey "M"> -<!ENTITY aboutTor.torbrowser_user_manual.label "Tor Browser Manual"> +<!ENTITY aboutTor.torbrowser_user_manual.label "Tor Browser-manual"> <!ENTITY aboutTor.tor_mission.label "The Tor Project is a US 501(c)(3) non-profit organization advancing human rights and freedoms by creating and deploying free and open source anonymity and privacy technologies, supporting their unrestricted availability and use, and furthering their scientific and popular understanding."> -<!ENTITY aboutTor.getInvolved.label "Get Involved »"> +<!ENTITY aboutTor.getInvolved.label "Deltag »"> <!ENTITY aboutTor.getInvolved.link "https://www.torproject.org/getinvolved/volunteer.html.en">

1 0

[metrics-web/master] Remove hidserv-frac-reporting graph and CSV file.
by karsten＠torproject.org 15 Aug '18

15 Aug '18

commit ec86ad02973963e9efad46b0ac21ae4f04c87fc6 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Aug 9 16:13:45 2018 +0200 Remove hidserv-frac-reporting graph and CSV file. This was the graph showing the "Fraction of relays reporting onion-service statistics". Fraction values will still be available as part of the other onion-service statistics' per-graph CSV files. Implements #26950. --- src/main/R/rserver/graphs.R | 34 ---------------------- src/main/resources/web.xml | 4 --- src/main/resources/web/json/categories.json | 3 +- src/main/resources/web/json/metrics.json | 11 ------- .../resources/web/jsps/reproducible-metrics.jsp | 1 - src/main/resources/web/jsps/stats.jsp | 25 ---------------- 6 files changed, 1 insertion(+), 77 deletions(-) diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R index ab37a32..66a1414 100644 --- a/src/main/R/rserver/graphs.R +++ b/src/main/R/rserver/graphs.R @@ -1287,40 +1287,6 @@ write_hidserv_rend_relayed_cells <- function(start_p = NULL, end_p = NULL, write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } -prepare_hidserv_frac_reporting <- function(start_p, end_p) { - read.csv(paste(stats_dir, "hidserv.csv", sep = ""), - colClasses = c("date" = "Date")) %>% - filter(if (!is.null(start_p)) date >= as.Date(start_p) else TRUE) %>% - filter(if (!is.null(end_p)) date <= as.Date(end_p) else TRUE) %>% - select(date, frac, type) -} - -plot_hidserv_frac_reporting <- function(start_p, end_p, path_p) { - prepare_hidserv_frac_reporting(start_p, end_p) %>% - ggplot(aes(x = date, y = frac, colour = type)) + - geom_line() + - geom_hline(yintercept = 0.01, linetype = 2) + - scale_x_date(name = "", breaks = custom_breaks, - labels = custom_labels, minor_breaks = custom_minor_breaks) + - scale_y_continuous(name = "", labels = percent, limits = c(0, NA)) + - scale_colour_hue(name = "", - breaks = c("rend-relayed-cells", "dir-onions-seen"), - labels = c("Onion-service traffic", - "Unique .onion addresses")) + - ggtitle(paste("Fraction of relays reporting onion-service", - "statistics")) + - labs(caption = copyright_notice) + - theme(legend.position = "top") - ggsave(filename = path_p, width = 8, height = 5, dpi = 150) -} - -write_hidserv_frac_reporting <- function(start_p = NULL, end_p = NULL, path_p) { - prepare_hidserv_frac_reporting(start_p, end_p) %>% - mutate(type = ifelse(type == "dir-onions-seen", "onions", "relayed")) %>% - spread(type, frac) %>% - write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") -} - prepare_webstats_tb <- function(start_p, end_p) { load(paste(rdata_dir, "webstats-tb.RData", sep = "")) data %>% diff --git a/src/main/resources/web.xml b/src/main/resources/web.xml index ae252da..5c88564 100644 --- a/src/main/resources/web.xml +++ b/src/main/resources/web.xml @@ -48,7 +48,6 @@ <url-pattern>/connbidirect.html</url-pattern> <url-pattern>/hidserv-dir-onions-seen.html</url-pattern> <url-pattern>/hidserv-rend-relayed-cells.html</url-pattern> - <url-pattern>/hidserv-frac-reporting.html</url-pattern> <url-pattern>/webstats-tb.html</url-pattern> <url-pattern>/webstats-tb-platform.html</url-pattern> <url-pattern>/webstats-tb-locale.html</url-pattern> @@ -190,9 +189,6 @@ <url-pattern>/hidserv-rend-relayed-cells.png</url-pattern> <url-pattern>/hidserv-rend-relayed-cells.pdf</url-pattern> <url-pattern>/hidserv-rend-relayed-cells.csv</url-pattern> - <url-pattern>/hidserv-frac-reporting.png</url-pattern> - <url-pattern>/hidserv-frac-reporting.pdf</url-pattern> - <url-pattern>/hidserv-frac-reporting.csv</url-pattern> <url-pattern>/webstats-tb.png</url-pattern> <url-pattern>/webstats-tb.pdf</url-pattern> <url-pattern>/webstats-tb.csv</url-pattern> diff --git a/src/main/resources/web/json/categories.json b/src/main/resources/web/json/categories.json index 23e6ee4..a5a4fcf 100644 --- a/src/main/resources/web/json/categories.json +++ b/src/main/resources/web/json/categories.json @@ -74,8 +74,7 @@ "description": "Onion services are services that are only accessible via the Tor network.", "metrics": [ "hidserv-dir-onions-seen", - "hidserv-rend-relayed-cells", - "hidserv-frac-reporting" + "hidserv-rend-relayed-cells" ] }, { diff --git a/src/main/resources/web/json/metrics.json b/src/main/resources/web/json/metrics.json index 0f85a28..ab5d98d 100644 --- a/src/main/resources/web/json/metrics.json +++ b/src/main/resources/web/json/metrics.json @@ -367,17 +367,6 @@ ] }, { - "id": "hidserv-frac-reporting", - "title": "Fraction of relays reporting onion-service statistics", - "type": "Graph", - "description": "<p>This graph shows the fraction of <a href=\"glossary.html#relay\">relays</a> that report statistics on <a href=\"glossary.html#onion-service\">onion service</a> usage. If at least 1% of relays report a statistic, it gets extrapolated towards a network total, where higher fractions are produce more accurate results. For more details on the extrapolation algorithm, see <a href=\"https://blog.torproject.org/blog/some-statistics-about-onions\">this blog post</a> and <a href=\"https://research.torproject.org/techreports/extrapolating-hidserv-stats-201…">this technical report</a>.</p>", - "function": "hidserv_frac_reporting", - "parameters": [ - "start", - "end" - ] - }, - { "id": "uncharted-data-flow", "title": "“TorFlow”", "type": "Link", diff --git a/src/main/resources/web/jsps/reproducible-metrics.jsp b/src/main/resources/web/jsps/reproducible-metrics.jsp index dc00751..87f692e 100644 --- a/src/main/resources/web/jsps/reproducible-metrics.jsp +++ b/src/main/resources/web/jsps/reproducible-metrics.jsp @@ -671,7 +671,6 @@ For further background on the following steps, refer to the technical report tit <ul> <li>Unique .onion addresses (version 2 only) <a href="/hidserv-dir-onions-seen.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a></li> <li>Onion-service traffic (versions 2 and 3) <a href="/hidserv-rend-relayed-cells.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a></li> -<li>Fraction of relays reporting onion-service statistics <a href="/hidserv-frac-reporting.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a></li> </ul> <h4>Step 1: Parse reported statistics from extra-info descriptors</h4> diff --git a/src/main/resources/web/jsps/stats.jsp b/src/main/resources/web/jsps/stats.jsp index 719176c..565f235 100644 --- a/src/main/resources/web/jsps/stats.jsp +++ b/src/main/resources/web/jsps/stats.jsp @@ -633,31 +633,6 @@ Onion Services <a href="#onion-services" name="servers" class="anchor">#</a></h2 <li><b>frac:</b> Total network fraction of statistics reported by rendezvous points.</li> </ul> -<h3>Fraction of relays reporting onion-service statistics -<a href="/hidserv-frac-reporting.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> -<a href="/hidserv-frac-reporting.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> -<a href="/hidserv-frac-reporting">#</a></h3> - -<h4>Parameters</h4> - -<ul> -<li><b>start:</b> First UTC date (YYYY-MM-DD) to include in the file.</li> -<li><b>end:</b> Last UTC date (YYYY-MM-DD) to include in the file.</li> -</ul> - -<h4>Columns</h4> - -<ul> -<li><b>date:</b> UTC date (YYYY-MM-DD) when relays have been listed as running.</li> -<li><b>onions:</b> Total network fraction of statistics reported by onion-service directories.</li> -<li><b>relayed:</b> Total network fraction of statistics reported by rendezvous points.</li> -</ul> - -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Remove section and data as suggested on <a href="https://bugs.torproject.org/26950">#26950</a>.</p> -</div> - </div> <div class="container">

1 0

[metrics-web/master] Make suggested changes to per-graph CSV files.
by karsten＠torproject.org 15 Aug '18

15 Aug '18

commit 0e6936003d0b9b4cc8d9388ea8a56eadcfdce1f9 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Aug 9 16:08:17 2018 +0200 Make suggested changes to per-graph CSV files. Implements #26998. --- src/main/R/clients/split-clients.R | 6 +- src/main/R/rserver/graphs.R | 70 ++++++---------- src/main/resources/web/jsps/stats.jsp | 153 ++++++++-------------------------- 3 files changed, 64 insertions(+), 165 deletions(-) diff --git a/src/main/R/clients/split-clients.R b/src/main/R/clients/split-clients.R index 50b03d4..9f80902 100644 --- a/src/main/R/clients/split-clients.R +++ b/src/main/R/clients/split-clients.R @@ -1,12 +1,12 @@ dir.create("RData", showWarnings = FALSE) c <- read.csv("clients.csv", stringsAsFactors = FALSE) -data <- c[c$node == 'relay', !(names(c) %in% c("node", "frac"))] +data <- c[c$node == 'relay', !(names(c) %in% c("node"))] save(data, file = "RData/clients-relay.RData") -data <- c[c$node == 'bridge', !(names(c) %in% c("node", "frac"))] +data <- c[c$node == 'bridge', !(names(c) %in% c("node"))] save(data, file = "RData/clients-bridge.RData") u <- read.csv("userstats-combined.csv", stringsAsFactors = FALSE) -data <- u[, !(names(u) %in% c("node", "version", "frac"))] +data <- u[, !(names(u) %in% c("node", "version"))] save(data, file = "RData/userstats-bridge-combined.RData") diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R index 12a80e9..ab37a32 100644 --- a/src/main/R/rserver/graphs.R +++ b/src/main/R/rserver/graphs.R @@ -423,7 +423,6 @@ plot_versions <- function(start_p, end_p, path_p) { write_versions <- function(start_p = NULL, end_p = NULL, path_p) { prepare_versions(start_p, end_p) %>% - spread(key = "version", value = "relays", fill = 0) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -459,6 +458,7 @@ plot_platforms <- function(start_p, end_p, path_p) { write_platforms <- function(start_p = NULL, end_p = NULL, path_p) { prepare_platforms(start_p, end_p) %>% + mutate(platform = tolower(platform)) %>% spread(platform, relays) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -607,8 +607,6 @@ plot_relayflags <- function(start_p, end_p, flag_p, path_p) { write_relayflags <- function(start_p = NULL, end_p = NULL, flag_p = NULL, path_p) { prepare_relayflags(start_p, end_p, flag_p) %>% - mutate(flag = tolower(flag)) %>% - spread(flag, relays) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -832,9 +830,6 @@ plot_connbidirect <- function(start_p, end_p, path_p) { write_connbidirect <- function(start_p = NULL, end_p = NULL, path_p) { prepare_connbidirect(start_p, end_p) %>% rename(q1 = X0.25, md = X0.5, q3 = X0.75) %>% - gather(variable, value, -(date:direction)) %>% - unite(temp, direction, variable) %>% - spread(temp, value) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -900,7 +895,8 @@ plot_userstats <- function(start_p, end_p, node_p, variable_p, value_p, events_p, path_p) { load(paste(rdata_dir, "clients-", node_p, ".RData", sep = "")) c <- data - u <- c[c$date >= start_p & c$date <= end_p, ] + u <- c[c$date >= start_p & c$date <= end_p, c("date", "country", "transport", + "version", "lower", "upper", "clients")] u <- rbind(u, data.frame(date = start_p, country = ifelse(variable_p == "country" & value_p != "all", value_p, ""), transport = ifelse(variable_p == "transport", value_p, ""), @@ -1053,8 +1049,7 @@ write_userstats_relay_country <- function(start_p = NULL, end_p = NULL, country == ifelse(country_p == "all", "", country_p) else TRUE) %>% filter(transport == "") %>% filter(version == "") %>% - mutate(downturns = clients < lower, upturns = clients > upper) %>% - select(date, country, clients, downturns, upturns, lower, upper) %>% + select(date, country, clients, lower, upper, frac) %>% rename(users = clients) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -1069,7 +1064,7 @@ write_userstats_bridge_country <- function(start_p = NULL, end_p = NULL, country == ifelse(country_p == "all", "", country_p) else TRUE) %>% filter(transport == "") %>% filter(version == "") %>% - select(date, country, clients) %>% + select(date, country, clients, frac) %>% rename(users = clients) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -1083,24 +1078,21 @@ write_userstats_bridge_transport <- function(start_p = NULL, end_p = NULL, filter(country == "") %>% filter(version == "") %>% filter(transport != "") %>% - select(date, transport, clients) + select(date, transport, clients, frac) if (is.null(transport_p) || "!<OR>" %in% transport_p) { n <- u %>% filter(transport != "<OR>") %>% - group_by(date) %>% + group_by(date, frac) %>% summarize(clients = sum(clients)) u <- rbind(u, data.frame(date = n$date, transport = "!<OR>", - clients = n$clients)) + clients = n$clients, frac = n$frac)) } u %>% filter(if (!is.null(transport_p)) transport %in% transport_p else TRUE) %>% - mutate(transport = ifelse(transport == "<OR>", "default_or_protocol", - ifelse(transport == "!<OR>", "any_pt", - ifelse(transport == "<??>", "unknown_pluggable_transports", - transport)))) %>% group_by(date, transport) %>% - select(date, transport, clients) %>% - spread(transport, clients) %>% + select(date, transport, clients, frac) %>% + rename(users = clients) %>% + arrange(date, transport) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -1113,7 +1105,7 @@ write_userstats_bridge_version <- function(start_p = NULL, end_p = NULL, filter(country == "") %>% filter(transport == "") %>% filter(if (!is.null(version_p)) version == version_p else TRUE) %>% - select(date, version, clients) %>% + select(date, version, clients, frac) %>% rename(users = clients) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -1159,12 +1151,8 @@ write_userstats_bridge_combined <- function(start_p = NULL, end_p = NULL, write_userstats_bridge_country(start_p, end_p, country_p, path_p) } else { prepare_userstats_bridge_combined(start_p, end_p, country_p) %>% - select(date, country, transport, low, high) %>% - mutate(transport = ifelse(transport == "<OR>", "default_or_protocol", - ifelse(transport == "<??>", "unknown_transport", transport))) %>% - gather(variable, value, -(date:transport)) %>% - unite(temp, transport, variable) %>% - spread(temp, value) %>% + select(date, country, transport, low, high, frac) %>% + arrange(date, country, transport) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } } @@ -1201,8 +1189,8 @@ plot_advbwdist_perc <- function(start_p, end_p, p_p, path_p) { write_advbwdist_perc <- function(start_p = NULL, end_p = NULL, p_p = NULL, path_p) { prepare_advbwdist_perc(start_p, end_p, p_p) %>% - unite(temp, variable, percentile) %>% - spread(temp, advbw) %>% + spread(variable, advbw) %>% + rename(p = percentile) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -1238,8 +1226,8 @@ plot_advbwdist_relay <- function(start_p, end_p, n_p, path_p) { write_advbwdist_relay <- function(start_p = NULL, end_p = NULL, n_p = NULL, path_p) { prepare_advbwdist_relay(start_p, end_p, n_p) %>% - unite(temp, variable, relay) %>% - spread(temp, advbw) %>% + spread(variable, advbw) %>% + rename(n = relay) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } @@ -1249,7 +1237,7 @@ prepare_hidserv_dir_onions_seen <- function(start_p, end_p) { filter(if (!is.null(start_p)) date >= as.Date(start_p) else TRUE) %>% filter(if (!is.null(end_p)) date <= as.Date(end_p) else TRUE) %>% filter(type == "dir-onions-seen") %>% - transmute(date = date, onions = ifelse(frac >= 0.01, wiqm, NA)) + transmute(date, onions = ifelse(frac >= 0.01, wiqm, NA), frac) } plot_hidserv_dir_onions_seen <- function(start_p, end_p, path_p) { @@ -1277,7 +1265,7 @@ prepare_hidserv_rend_relayed_cells <- function(start_p, end_p) { filter(if (!is.null(end_p)) date <= as.Date(end_p) else TRUE) %>% filter(type == "rend-relayed-cells") %>% transmute(date, - relayed = ifelse(frac >= 0.01, wiqm * 8 * 512 / (86400 * 1e9), NA)) + relayed = ifelse(frac >= 0.01, wiqm * 8 * 512 / (86400 * 1e9), NA), frac) } plot_hidserv_rend_relayed_cells <- function(start_p, end_p, path_p) { @@ -1440,22 +1428,14 @@ plot_webstats_tb_locale <- function(start_p, end_p, path_p) { # plot_webstats_tb_locale needs the preliminary data frame e for its # breaks and labels. Left as future work. write_webstats_tb_locale <- function(start_p = NULL, end_p = NULL, path_p) { - d <- read.csv(paste(stats_dir, "webstats.csv", sep = ""), - colClasses = c("log_date" = "Date", "locale" = "character")) - d <- d %>% + read.csv(paste(stats_dir, "webstats.csv", sep = ""), + colClasses = c("log_date" = "Date", "locale" = "character")) %>% filter(if (!is.null(start_p)) log_date >= as.Date(start_p) else TRUE) %>% filter(if (!is.null(end_p)) log_date <= as.Date(end_p) else TRUE) %>% - filter(request_type == "tbid") - e <- d - e <- aggregate(list(count = e$count), by = list(locale = e$locale), FUN = sum) - e <- e[order(e$count, decreasing = TRUE), ] - e <- e[1:5, ] - d <- aggregate(list(count = d$count), by = list(log_date = d$log_date, - locale = ifelse(d$locale %in% e$locale, d$locale, "other")), FUN = sum) - d %>% - mutate(locale = tolower(locale)) %>% + filter(request_type == "tbid") %>% + group_by(log_date, locale) %>% + summarize(initial_downloads = sum(count)) %>% rename(date = log_date) %>% - spread(locale, count) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } diff --git a/src/main/resources/web/jsps/stats.jsp b/src/main/resources/web/jsps/stats.jsp index 1c18921..719176c 100644 --- a/src/main/resources/web/jsps/stats.jsp +++ b/src/main/resources/web/jsps/stats.jsp @@ -45,7 +45,7 @@ https://metrics.torproject.org/identifier.csv <li><b>February 28, 2018:</b> Added per-graph CSV files to eventually replace pre-aggregated CSV files.</li> <li><b>May 29, 2018:</b> Made all parameters of per-graph CSV files optional to support providing both pre-filtered and complete data sets.</li> <li><b>July 31, 2018:</b> Announced pending changes to per-graph CSV files to become effective on August 15 and pre-aggregated CSV files to be removed by September 15.</li> -<li><b>August 15, 2018 (scheduled):</b> Make the first batch of changes to per-graph CSV files (marked as "Suggested change" below).</li> +<li><b>August 15, 2018:</b> Made the first batch of changes to per-graph CSV files.</li> <li><b>September 15, 2018 (scheduled):</b> Remove all pre-aggregated CSV files.</li> </ul> @@ -75,22 +75,11 @@ Users <a href="#users" name="users" class="anchor">#</a></h2> <li><b>date:</b> UTC date (YYYY-MM-DD) for which user numbers are estimated.</li> <li><b>country:</b> Two-letter lower-case country code as found in a GeoIP database by resolving clients' IP addresses, or <b>"??"</b> if client IP addresses could not be resolved. If this column contains the empty string, all clients are included, regardless of their country code.</li> <li><b>users:</b> Estimated number of clients.</li> -<li><b>downturns:</b> Whether the estimated number of clients is below the lower number of expected clients, indicating a possible censorship-related event. If this column contains the empty string, there are no expectations on the number of clients.</li> -<li><b>upturns:</b> Whether the estimated number of clients is above the upper number of expected clients, indicating a possible censorship-related event. If this column contains the empty string, there are no expectations on the number of clients.</li> -<li><b>lower:</b> Lower number of expected clients under the assumption that there has been no censorship event. If this column contains the empty string, there are no expectations on the number of clients.</li> -<li><b>upper:</b> Upper number of expected clients under the assumption that there has been no release of censorship. If this column contains the empty string, there are no expectations on the number of clients.</li> +<li><b>lower:</b> Lower number of expected clients under the assumption that there has been no censorship event. If <b>users < lower</b>, a censorship-related event might have happened in this country on the given day. If this column contains the empty string, there are no expectations on the number of clients.</li> +<li><b>upper:</b> Upper number of expected clients under the assumption that there has been no release of censorship. If <b>users > upper</b>, a censorship-related event might have happened in this country on the given day. If this column contains the empty string, there are no expectations on the number of clients.</li> +<li><b>frac:</b> Fraction of relays in percent that the estimate is based on.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Remove the <b>downturns</b> and <b>upturns</b> columns which are trivial to compute as <b>users < lower</b> and <b>users > upper</b>, and which don't necessarily make the CSV file easier to handle. There could even be a gentle hint on computing the dots in the graph from two columns.</p> -</div> - -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Add <b>frac</b> column ("Fraction of relays (as value between 0 and 1) that the estimate is based on.") which might be relevant for pro users. Related to the discussion on <a href="https://bugs.torproject.org/26950">#26950</a>.</p> -</div> - <h3>Bridge users by country <a href="/userstats-bridge-country.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/userstats-bridge-country.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -110,13 +99,9 @@ Users <a href="#users" name="users" class="anchor">#</a></h2> <li><b>date:</b> UTC date (YYYY-MM-DD) for which user numbers are estimated.</li> <li><b>country:</b> Two-letter lower-case country code as found in a GeoIP database by resolving clients' IP addresses, or <b>"??"</b> if client IP addresses could not be resolved. If this column contains the empty string, all clients are included, regardless of their country code.</li> <li><b>users:</b> Estimated number of clients.</li> +<li><b>frac:</b> Fraction of bridges in percent that the estimate is based on.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Add <b>frac</b> column ("Fraction of bridges (as value between 0 and 1) that the estimate is based on.") which might be relevant for pro users. Related to the discussion on <a href="https://bugs.torproject.org/26950">#26950</a>.</p> -</div> - <h3>Bridge users by transport <a href="/userstats-bridge-transport.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/userstats-bridge-transport.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -134,19 +119,11 @@ Users <a href="#users" name="users" class="anchor">#</a></h2> <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) for which user numbers are estimated.</li> -<li><b>$transport:</b> One or more columns with the estimated number of clients using transport with lower-case name <b>$transport</b> to connect to the Tor network using bridges. Examples for transport names are <b>"obfs4"</b>, <b>"websocket"</b> for Flash proxy/websocket, <b>"fte"</b> for FTE, <b>"any_pt"</b> for any pluggable transport, <b>"unknown_pluggable_transports"</b> for unknown pluggable transport(s), or <b>"default_or_protocol"</b> for the default OR protocol.</li> +<li><b>transport:</b> Transport name used by clients to connect to the Tor network using bridges. Examples are <b>"obfs4"</b>, <b>"websocket"</b> for Flash proxy/websocket, <b>"fte"</b> for FTE, <b>"!<OR>"</b> for any pluggable transport, <b>"<??>"</b> for unknown pluggable transport(s), or <b>"<OR>"</b> for the default OR protocol.</li> +<li><b>users:</b> Estimated number of clients.</li> +<li><b>frac:</b> Fraction of bridges in percent that the estimate is based on.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Replace <b>$transport</b> by a <b>transport</b> column for the transport name (written as non-percent-encoded !<OR>, <??>, and <OR> for consistency with the <b>transport</b> parameter) and a <b>users</b> column for the estimated number of clients, similar to the bridge users by country graph.</p> -</div> - -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Add <b>frac</b> column ("Fraction of bridges (as value between 0 and 1) that the estimate is based on.") which might be relevant for pro users. Related to the discussion on <a href="https://bugs.torproject.org/26950">#26950</a>.</p> -</div> - <h3>Bridge users by country and transport <a href="/userstats-bridge-combined.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/userstats-bridge-combined.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -165,20 +142,12 @@ Users <a href="#users" name="users" class="anchor">#</a></h2> <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) for which user numbers are estimated.</li> <li><b>country:</b> Two-letter lower-case country code as found in a GeoIP database by resolving clients' IP addresses, or <b>"??"</b> if client IP addresses could not be resolved.</li> -<li><b>$transport_high:</b> Upper bound of estimated users from the given country and transport. Transport names are written in lower case, and the default OR protocol is written as <b>default_or_protocol</b>.</li> -<li><b>$transport_low:</b> Lower bound of estimated users from the given country and transport. Transport names are written in lower case, and the default OR protocol is written as <b>default_or_protocol</b>.</li> +<li><b>transport:</b> Transport name used by clients to connect to the Tor network using bridges. Examples are <b>"obfs4"</b>, <b>"websocket"</b> for Flash proxy/websocket, <b>"fte"</b> for FTE, <b>"<??>"</b> for unknown pluggable transport(s), or <b>"<OR>"</b> for the default OR protocol.</li> +<li><b>high:</b> Upper bound of estimated users from the given country and transport.</li> +<li><b>low:</b> Lower bound of estimated users from the given country and transport.</li> +<li><b>frac:</b> Fraction of bridges in percent that the estimate is based on.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Replace <b>$transport_high</b> and <b>$transport_low</b> by a <b>transport</b> column for the transport name (written as non-percent-encoded <OR> for consistency with the previous graph) and a <b>high</b> and a <b>low</b> column for the upper and lower bound.</p> -</div> - -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Add <b>frac</b> column ("Fraction of bridges (as value between 0 and 1) that the estimate is based on.") which might be relevant for pro users. Related to the discussion on <a href="https://bugs.torproject.org/26950">#26950</a>.</p> -</div> - <h3>Bridge users by IP version <a href="/userstats-bridge-version.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/userstats-bridge-version.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -199,15 +168,9 @@ using bridges, which can be either <b>"v4"</b> or <b>"v6"</b>.</li> <li><b>date:</b> UTC date (YYYY-MM-DD) for which user numbers are estimated.</li> <li><b>version:</b> IP version used by clients to connect to the Tor network using bridges, which can be either <b>"v4"</b> or <b>"v6"</b>. If this column contains the empty string, all clients are included, regardless of their IP version.</li> <li><b>users:</b> Estimated number of clients.</li> +<li><b>frac:</b> Fraction of bridges in percent that the estimate is based on.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Add <b>frac</b> column ("Fraction of bridges (as value between 0 and 1) that the estimate is based on.") which might be relevant for pro users. Related to the discussion on <a href="https://bugs.torproject.org/26950">#26950</a>.</p> -</div> - -</div> - <div class="container"> <h2><i class="fa fa-server fa-fw" aria-hidden="true"></i> Servers <a href="#servers" name="servers" class="anchor">#</a></h2> @@ -249,14 +212,10 @@ Servers <a href="#servers" name="servers" class="anchor">#</a></h2> <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) when relays have been listed as running.</li> -<li><b>$flag:</b> Average number of relays with the given relay flag in lower case, which can be <b>"exit"</b>, <b>"fast"</b>, <b>"guard"</b>, <b>"hsdir"</b>, <b>"fast"</b>, <b>"running"</b>, and <b>"stable"</b>.</li> +<li><b>flag:</b> Relay flag, which can be <b>"Exit"</b>, <b>"Fast"</b>, <b>"Guard"</b>, <b>"HSDir"</b>, <b>"Fast"</b>, <b>"Running"</b>, or <b>"Stable"</b>.</li> +<li><b>relays:</b> Average number of relays.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Replace <b>$flag</b> columns by a <b>flag</b> and a <b>relays</b> column, and include the relay flag name in their original capitalization, rather than lower-cased.</p> -</div> - <h3>Relays by tor version <a href="/versions.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/versions.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -273,14 +232,10 @@ Servers <a href="#servers" name="servers" class="anchor">#</a></h2> <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) when relays have been listed as running.</li> -<li><b>$version:</b> Average number of relays with the given first three dotted numbers of the Tor software version as reported by the relay. An example is <b>"0.3.4"</b>. +<li><b>version:</b> First three dotted numbers of the Tor software version as reported by the relay. An example is <b>"0.3.4"</b>.</li> +<li><b>relays:</b> Average number of relays.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Replace <b>$version</b> columns by a <b>version</b> and a <b>relays</b> column.</p> -</div> - <h3>Relays by platform <a href="/platforms.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/platforms.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -297,18 +252,13 @@ Servers <a href="#servers" name="servers" class="anchor">#</a></h2> <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) when relays have been listed as running.</li> -<li><b>BSD:</b> Average number of relays on *BSD.</li> -<li><b>Linux:</b> Average number of relays on Linux.</li> -<li><b>Other:</b> Average number of relays on another platform than Linux, *BSD, Windows, or macOS.</li> -<li><b>Windows:</b> Average number of relays on Windows.</li> -<li><b>macOS:</b> Average number of relays on macOS.</li> +<li><b>bsd:</b> Average number of relays on *BSD.</li> +<li><b>linux:</b> Average number of relays on Linux.</li> +<li><b>macos:</b> Average number of relays on macOS.</li> +<li><b>other:</b> Average number of relays on another platform than Linux, *BSD, Windows, or macOS.</li> +<li><b>windows:</b> Average number of relays on Windows.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Change platform-specific columns to be all lower-case as a good practice to only use lower-cased column names everywhere.</p> -</div> - <h3>Relays by IP version <a href="/relays-ipv6.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/relays-ipv6.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -440,14 +390,9 @@ Traffic <a href="#traffic" name="traffic" class="anchor">#</a></h2> <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) that relays reported bandwidth data for.</li> -<li><b>all_$p:</b> Advertised bandwidth in Gbit/s of the p-th percentile of all relays.</li> -<li><b>exits_$p:</b> Advertised bandwidth in Gbit/s of the p-th percentile of relays with the <b>"Exit"</b> relay flag.</li> -</ul> - -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Replace <b>all_p$</b> and <b>exits_$p</b> columns by three columns <b>p</b>, <b>all</b>, and <b>exit</b>.</p> -</div> +<li><b>p:</b> Percentile as value between 0 and 100.</li> +<li><b>all:</b> Advertised bandwidth in Gbit/s of the p-th percentile of all relays.</li> +<li><b>exits:</b> Advertised bandwidth in Gbit/s of the p-th percentile of relays with the <b>"Exit"</b> relay flag.</li> <h3>Advertised bandwidth of n-th fastest relays <a href="/advbwdist-relay.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> @@ -466,15 +411,11 @@ Traffic <a href="#traffic" name="traffic" class="anchor">#</a></h2> <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) that relays reported bandwidth data for.</li> -<li><b>all_$n:</b> Advertised bandwidth in Gbit/s of n-th fastest relay.</li> -<li><b>exits_$n:</b> Advertised bandwidth in Gbit/s of n-th fastest relay with the <b>"Exit"</b> relay flag.</li> +<li><b>n:</b> Position of the relay in an ordered list of all advertised bandwidths, starting at 1 for the fastest relay in the network.</li> +<li><b>all:</b> Advertised bandwidth in Gbit/s of n-th fastest relay.</li> +<li><b>exits:</b> Advertised bandwidth in Gbit/s of n-th fastest relay with the <b>"Exit"</b> relay flag.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Replace <b>all_n$</b> and <b>exits_$n</b> columns by three columns <b>n</b>, <b>all</b>, and <b>exit</b>.</p> -</div> - <h3>Consumed bandwidth by Exit/Guard flag combination <a href="/bwhist-flags.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/bwhist-flags.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -533,22 +474,12 @@ Traffic <a href="#traffic" name="traffic" class="anchor">#</a></h2> <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) for which statistics on uni-/bidirectional connection usage were reported.</li> -<li><b>both_md:</b> Median of fraction of connections classified as both reading and writing.</li> -<li><b>both_q1:</b> First quartile of fraction of connections classified as both reading and writing.</li> -<li><b>both_q3:</b> Third quartile of fraction of connections classified as both reading and writing.</li> -<li><b>read_md:</b> Median of fraction of connections classified as mostly reading.</li> -<li><b>read_q1:</b> First quartile of fraction of connections classified as mostly reading.</li> -<li><b>read_q3:</b> Third quartile of fraction of connections classified as mostly reading.</li> -<li><b>write_md:</b> Median of fraction of connections classified as mostly writing.</li> -<li><b>write_q1:</b> First quartile of fraction of connections classified as mostly writing.</li> -<li><b>write_q3:</b> Third quartile of fraction of connections classified as mostly writing.</li> +<li><b>direction:</b> Direction of reported fraction, which can be <b>"read"</b>, <b>"write"</b>, or <b>"both"</b> for connections classified as "mostly reading", "mostly writing", or "both reading and writing". Connections below the threshold have been removed from this statistics file entirely.</li> +<li><b>q1:</b> First quartile of fraction of connections.</li> +<li><b>md:</b> Median of fraction of connections.</li> +<li><b>q3:</b> Third quartile of fraction of connections.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Replace columns except <b>date</b> by four columns <b>direction</b>, <b>q1</b>, <b>md</b>, and <b>q3</b>.</p> -</div> - </div> <div class="container"> @@ -679,13 +610,9 @@ Onion Services <a href="#onion-services" name="servers" class="anchor">#</a></h2 <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) when relays have been listed as running.</li> <li><b>onions:</b> Estimated number of unique .onion addresses observed by onion-service directories.</li> +<li><b>frac:</b> Total network fraction of statistics reported by onion-service directories.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Add <b>frac</b> column as suggested on <a href="https://bugs.torproject.org/26950">#26950</a>.</p> -</div> - <h3>Onion-service traffic (versions 2 and 3) <a href="/hidserv-rend-relayed-cells.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/hidserv-rend-relayed-cells.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -703,13 +630,9 @@ Onion Services <a href="#onion-services" name="servers" class="anchor">#</a></h2 <ul> <li><b>date:</b> UTC date (YYYY-MM-DD) when relays have been listed as running.</li> <li><b>relayed:</b> Estimated bandwidth in Gbit/s relayed on rendezvous circuits as observed by rendezvous points.</li> +<li><b>frac:</b> Total network fraction of statistics reported by rendezvous points.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Add <b>frac</b> column as suggested on <a href="https://bugs.torproject.org/26950">#26950</a>.</p> -</div> - <h3>Fraction of relays reporting onion-service statistics <a href="/hidserv-frac-reporting.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/hidserv-frac-reporting.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a> @@ -800,14 +723,10 @@ Applications <a href="#applications" name="applications" class="anchor">#</a></h </ul> <li><b>date:</b> UTC date (YYYY-MM-DD) when requests to <code>torproject.org</code> web servers have been logged.</li> -<li><b>$locale:</b> Number of Tor Browser initial downloads for the given locale; limited to the top-5 locales in the requested time period.</li> +<li><b>locale:</b> Locale, like "en-US" for English (United States), "de" for German, etc., and "??" for unrecognized locales.</li> +<li><b>initial_downloads:</b> Number of Tor Browser initial downloads.</li> </ul> -<div class="bs-callout bs-callout-warning"> -<h3>Suggested change</h3> -<p>Replace all locale-specific columns by two columns <b>locale</b> and <b>count</b> to avoid dynamically changing columns. Maybe also take out the limitation to top-5 locales in the file (not the graph), similar to how the "Bridge users by country and transport" file contains all transports, not just the top-3 ones.</p> -</div> - <h3>Tor Messenger downloads and updates <a href="/webstats-tm.html" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> graph</a> <a href="/webstats-tm.csv" class="btn btn-primary btn-xs"><i class="fa fa-chevron-right" aria-hidden="true"></i> data</a>

1 0

[tor/release-0.3.3] Detect broken stdatomic.h, and pretend that it isn't there at all
by nickm＠torproject.org 15 Aug '18

15 Aug '18

commit b66386865e68d8df0f228187781fcd5bc2dbf184 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Aug 8 15:41:40 2018 -0400 Detect broken stdatomic.h, and pretend that it isn't there at all I hope that the debian clang maintainers will look at debian bug 903709 soon. But until they do, this should keep our users and our CI happy on sid with clang. Closes ticket 26779. --- changes/bug26779 | 4 ++++ configure.ac | 20 ++++++++++++++++++++ src/common/compat_threads.c | 5 ++--- src/common/compat_threads.h | 19 +++++++++++-------- 4 files changed, 37 insertions(+), 11 deletions(-) diff --git a/changes/bug26779 b/changes/bug26779 new file mode 100644 index 000000000..fb7f6160e --- /dev/null +++ b/changes/bug26779 @@ -0,0 +1,4 @@ + o Minor features (bug workaround): + - Compile correctly on systems that provide the C11 stdatomic.h header, + but where C11 atomic functions don't actually compile. + Closes ticket 26779; workaround for Debian issue 903709. diff --git a/configure.ac b/configure.ac index cd591c785..8e6683bb7 100644 --- a/configure.ac +++ b/configure.ac @@ -1557,6 +1557,26 @@ AC_CHECK_SIZEOF(socklen_t, , [AC_INCLUDES_DEFAULT() AC_CHECK_SIZEOF(cell_t) +# Let's see if stdatomic works. (There are some debian clangs that screw it +# up; see Tor bug #26779 and debian bug 903709.) +AC_CACHE_CHECK([whether C11 stdatomic.h actually works], + tor_cv_stdatomic_works, +[AC_COMPILE_IFELSE([AC_LANG_SOURCE([[ +#include <stdatomic.h> +struct x { atomic_size_t y; }; +void try_atomic_init(struct x *xx) +{ + atomic_init(&xx->y, 99); + atomic_fetch_add(&xx->y, 1); +} +]])], [tor_cv_stdatomic_works=yes], [tor_cv_stdatomic_works=no])]) + +if test "$tor_cv_stdatomic_works" = "yes"; then + AC_DEFINE(STDATOMIC_WORKS, 1, [Set to 1 if we can compile a simple stdatomic example.]) +elif test "$ac_cv_header_stdatomic_h" = "yes"; then + AC_MSG_WARN([Your compiler provides the stdatomic.h header, but it doesn't seem to work. I'll pretend it isn't there. If you are using Clang on Debian, maybe this is because of https://bugs.debian.org/903709 ]) +fi + # Now make sure that NULL can be represented as zero bytes. AC_CACHE_CHECK([whether memset(0) sets pointers to NULL], tor_cv_null_is_zero, [AC_RUN_IFELSE([AC_LANG_SOURCE( diff --git a/src/common/compat_threads.c b/src/common/compat_threads.c index 3171c4b2f..9f64c0634 100644 --- a/src/common/compat_threads.c +++ b/src/common/compat_threads.c @@ -352,7 +352,7 @@ alert_sockets_close(alert_sockets_t *socks) socks->read_fd = socks->write_fd = -1; } -#ifndef HAVE_STDATOMIC_H +#ifndef HAVE_WORKING_STDATOMIC /** Initialize a new atomic counter with the value 0 */ void atomic_counter_init(atomic_counter_t *counter) @@ -403,5 +403,4 @@ atomic_counter_exchange(atomic_counter_t *counter, size_t newval) tor_mutex_release(&counter->mutex); return oldval; } -#endif /* !defined(HAVE_STDATOMIC_H) */ - +#endif /* !defined(HAVE_WORKING_STDATOMIC) */ diff --git a/src/common/compat_threads.h b/src/common/compat_threads.h index c93e601ec..8bf822568 100644 --- a/src/common/compat_threads.h +++ b/src/common/compat_threads.h @@ -14,7 +14,11 @@ #include <pthread.h> #endif -#ifdef HAVE_STDATOMIC_H +#if defined(HAVE_STDATOMIC_H) && defined(STDATOMIC_WORKS) +#define HAVE_WORKING_STDATOMIC +#endif + +#ifdef HAVE_WORKING_STDATOMIC #include <stdatomic.h> #endif @@ -156,18 +160,18 @@ void tor_threadlocal_set(tor_threadlocal_t *threadlocal, void *value); /** * Atomic counter type; holds a size_t value. */ -#ifdef HAVE_STDATOMIC_H +#ifdef HAVE_WORKING_STDATOMIC typedef struct atomic_counter_t { atomic_size_t val; } atomic_counter_t; #define ATOMIC_LINKAGE static -#else /* !(defined(HAVE_STDATOMIC_H)) */ +#else /* !(defined(HAVE_WORKING_STDATOMIC)) */ typedef struct atomic_counter_t { tor_mutex_t mutex; size_t val; } atomic_counter_t; #define ATOMIC_LINKAGE -#endif /* defined(HAVE_STDATOMIC_H) */ +#endif /* defined(HAVE_WORKING_STDATOMIC) */ ATOMIC_LINKAGE void atomic_counter_init(atomic_counter_t *counter); ATOMIC_LINKAGE void atomic_counter_destroy(atomic_counter_t *counter); @@ -178,7 +182,7 @@ ATOMIC_LINKAGE size_t atomic_counter_exchange(atomic_counter_t *counter, size_t newval); #undef ATOMIC_LINKAGE -#ifdef HAVE_STDATOMIC_H +#ifdef HAVE_WORKING_STDATOMIC /** Initialize a new atomic counter with the value 0 */ static inline void atomic_counter_init(atomic_counter_t *counter) @@ -216,8 +220,7 @@ atomic_counter_exchange(atomic_counter_t *counter, size_t newval) return atomic_exchange(&counter->val, newval); } -#else /* !(defined(HAVE_STDATOMIC_H)) */ -#endif /* defined(HAVE_STDATOMIC_H) */ +#else /* !(defined(HAVE_WORKING_STDATOMIC)) */ +#endif /* defined(HAVE_WORKING_STDATOMIC) */ #endif /* !defined(TOR_COMPAT_THREADS_H) */ -

1 0

[tor/release-0.3.3] Merge remote-tracking branch 'public/bug26779_033' into maint-0.3.3
by nickm＠torproject.org 15 Aug '18

15 Aug '18

commit 1868982de6c538b75ab599b332fa896492608620 Merge: fde551a38 b66386865 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Aug 15 02:16:19 2018 -0400 Merge remote-tracking branch 'public/bug26779_033' into maint-0.3.3 changes/bug26779 | 4 ++++ configure.ac | 20 ++++++++++++++++++++ src/common/compat_threads.c | 5 ++--- src/common/compat_threads.h | 19 +++++++++++-------- 4 files changed, 37 insertions(+), 11 deletions(-)

1 0

[tor/release-0.3.3] Merge branch 'maint-0.3.3' into release-0.3.3
by nickm＠torproject.org 15 Aug '18

15 Aug '18

commit ef4d5d99fb53a313e94b4bf34a4d0ae2f7ba0fef Merge: b2fa807b9 1868982de Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Aug 15 02:18:54 2018 -0400 Merge branch 'maint-0.3.3' into release-0.3.3 changes/bug26779 | 4 ++++ configure.ac | 20 ++++++++++++++++++++ src/common/compat_threads.c | 5 ++--- src/common/compat_threads.h | 19 +++++++++++-------- 4 files changed, 37 insertions(+), 11 deletions(-)

1 0

[tor/release-0.3.4] Detect broken stdatomic.h, and pretend that it isn't there at all
by nickm＠torproject.org 15 Aug '18

15 Aug '18

commit b66386865e68d8df0f228187781fcd5bc2dbf184 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Aug 8 15:41:40 2018 -0400 Detect broken stdatomic.h, and pretend that it isn't there at all I hope that the debian clang maintainers will look at debian bug 903709 soon. But until they do, this should keep our users and our CI happy on sid with clang. Closes ticket 26779. --- changes/bug26779 | 4 ++++ configure.ac | 20 ++++++++++++++++++++ src/common/compat_threads.c | 5 ++--- src/common/compat_threads.h | 19 +++++++++++-------- 4 files changed, 37 insertions(+), 11 deletions(-) diff --git a/changes/bug26779 b/changes/bug26779 new file mode 100644 index 000000000..fb7f6160e --- /dev/null +++ b/changes/bug26779 @@ -0,0 +1,4 @@ + o Minor features (bug workaround): + - Compile correctly on systems that provide the C11 stdatomic.h header, + but where C11 atomic functions don't actually compile. + Closes ticket 26779; workaround for Debian issue 903709. diff --git a/configure.ac b/configure.ac index cd591c785..8e6683bb7 100644 --- a/configure.ac +++ b/configure.ac @@ -1557,6 +1557,26 @@ AC_CHECK_SIZEOF(socklen_t, , [AC_INCLUDES_DEFAULT() AC_CHECK_SIZEOF(cell_t) +# Let's see if stdatomic works. (There are some debian clangs that screw it +# up; see Tor bug #26779 and debian bug 903709.) +AC_CACHE_CHECK([whether C11 stdatomic.h actually works], + tor_cv_stdatomic_works, +[AC_COMPILE_IFELSE([AC_LANG_SOURCE([[ +#include <stdatomic.h> +struct x { atomic_size_t y; }; +void try_atomic_init(struct x *xx) +{ + atomic_init(&xx->y, 99); + atomic_fetch_add(&xx->y, 1); +} +]])], [tor_cv_stdatomic_works=yes], [tor_cv_stdatomic_works=no])]) + +if test "$tor_cv_stdatomic_works" = "yes"; then + AC_DEFINE(STDATOMIC_WORKS, 1, [Set to 1 if we can compile a simple stdatomic example.]) +elif test "$ac_cv_header_stdatomic_h" = "yes"; then + AC_MSG_WARN([Your compiler provides the stdatomic.h header, but it doesn't seem to work. I'll pretend it isn't there. If you are using Clang on Debian, maybe this is because of https://bugs.debian.org/903709 ]) +fi + # Now make sure that NULL can be represented as zero bytes. AC_CACHE_CHECK([whether memset(0) sets pointers to NULL], tor_cv_null_is_zero, [AC_RUN_IFELSE([AC_LANG_SOURCE( diff --git a/src/common/compat_threads.c b/src/common/compat_threads.c index 3171c4b2f..9f64c0634 100644 --- a/src/common/compat_threads.c +++ b/src/common/compat_threads.c @@ -352,7 +352,7 @@ alert_sockets_close(alert_sockets_t *socks) socks->read_fd = socks->write_fd = -1; } -#ifndef HAVE_STDATOMIC_H +#ifndef HAVE_WORKING_STDATOMIC /** Initialize a new atomic counter with the value 0 */ void atomic_counter_init(atomic_counter_t *counter) @@ -403,5 +403,4 @@ atomic_counter_exchange(atomic_counter_t *counter, size_t newval) tor_mutex_release(&counter->mutex); return oldval; } -#endif /* !defined(HAVE_STDATOMIC_H) */ - +#endif /* !defined(HAVE_WORKING_STDATOMIC) */ diff --git a/src/common/compat_threads.h b/src/common/compat_threads.h index c93e601ec..8bf822568 100644 --- a/src/common/compat_threads.h +++ b/src/common/compat_threads.h @@ -14,7 +14,11 @@ #include <pthread.h> #endif -#ifdef HAVE_STDATOMIC_H +#if defined(HAVE_STDATOMIC_H) && defined(STDATOMIC_WORKS) +#define HAVE_WORKING_STDATOMIC +#endif + +#ifdef HAVE_WORKING_STDATOMIC #include <stdatomic.h> #endif @@ -156,18 +160,18 @@ void tor_threadlocal_set(tor_threadlocal_t *threadlocal, void *value); /** * Atomic counter type; holds a size_t value. */ -#ifdef HAVE_STDATOMIC_H +#ifdef HAVE_WORKING_STDATOMIC typedef struct atomic_counter_t { atomic_size_t val; } atomic_counter_t; #define ATOMIC_LINKAGE static -#else /* !(defined(HAVE_STDATOMIC_H)) */ +#else /* !(defined(HAVE_WORKING_STDATOMIC)) */ typedef struct atomic_counter_t { tor_mutex_t mutex; size_t val; } atomic_counter_t; #define ATOMIC_LINKAGE -#endif /* defined(HAVE_STDATOMIC_H) */ +#endif /* defined(HAVE_WORKING_STDATOMIC) */ ATOMIC_LINKAGE void atomic_counter_init(atomic_counter_t *counter); ATOMIC_LINKAGE void atomic_counter_destroy(atomic_counter_t *counter); @@ -178,7 +182,7 @@ ATOMIC_LINKAGE size_t atomic_counter_exchange(atomic_counter_t *counter, size_t newval); #undef ATOMIC_LINKAGE -#ifdef HAVE_STDATOMIC_H +#ifdef HAVE_WORKING_STDATOMIC /** Initialize a new atomic counter with the value 0 */ static inline void atomic_counter_init(atomic_counter_t *counter) @@ -216,8 +220,7 @@ atomic_counter_exchange(atomic_counter_t *counter, size_t newval) return atomic_exchange(&counter->val, newval); } -#else /* !(defined(HAVE_STDATOMIC_H)) */ -#endif /* defined(HAVE_STDATOMIC_H) */ +#else /* !(defined(HAVE_WORKING_STDATOMIC)) */ +#endif /* defined(HAVE_WORKING_STDATOMIC) */ #endif /* !defined(TOR_COMPAT_THREADS_H) */ -

1 0

[tor/release-0.3.4] Merge remote-tracking branch 'public/bug26779_033' into maint-0.3.3
by nickm＠torproject.org 15 Aug '18

15 Aug '18

commit 1868982de6c538b75ab599b332fa896492608620 Merge: fde551a38 b66386865 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Aug 15 02:16:19 2018 -0400 Merge remote-tracking branch 'public/bug26779_033' into maint-0.3.3 changes/bug26779 | 4 ++++ configure.ac | 20 ++++++++++++++++++++ src/common/compat_threads.c | 5 ++--- src/common/compat_threads.h | 19 +++++++++++-------- 4 files changed, 37 insertions(+), 11 deletions(-)

1 0

[tor/release-0.3.4] Merge branch 'maint-0.3.3' into maint-0.3.4
by nickm＠torproject.org 15 Aug '18

15 Aug '18

commit 6c1d2549df546b6d2628582c9b078753eecd16f4 Merge: b4362b99c 1868982de Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Aug 15 02:16:23 2018 -0400 Merge branch 'maint-0.3.3' into maint-0.3.4 changes/bug26779 | 4 ++++ configure.ac | 20 ++++++++++++++++++++ src/common/compat_threads.c | 5 ++--- src/common/compat_threads.h | 19 +++++++++++-------- 4 files changed, 37 insertions(+), 11 deletions(-)

1 0