morgan pushed to branch main at The Tor Project / Applications / tor-browser-spec
Commits: 1472857c by Richard Pospesel at 2024-06-27T04:18:44+00:00 Create bugzilla2gitlab script for ESR resolved issue audits
- fetches all resolved bugs for a firefox release - outputs gitlab markdown for each entry which: - displays bugzilla issue number, title - links to bugzilla issue - shows a button which when clicked populates a review issue prepopulated with: - bugzilla information - appropriate gitlab labels - links to parent audit issue - provides checklist for engineers to mark blocks as triaged
- - - - - aaf00ad7 by Morgan at 2024-10-22T18:49:55+00:00 updated code_audit.sh script to handle .mjs js files and some minor tweaks
- - - - - d3418425 by Morgan at 2024-10-22T18:50:15+00:00 FF116-FF128 Audits
- - - - -
15 changed files:
- + audits/FF116_AUDIT - + audits/FF117_AUDIT - + audits/FF118_AUDIT - + audits/FF119_AUDIT - + audits/FF120_AUDIT - + audits/FF121_AUDIT - + audits/FF122_AUDIT - + audits/FF123_AUDIT - + audits/FF124_AUDIT - + audits/FF125_AUDIT - + audits/FF126_AUDIT - + audits/FF127_AUDIT - + audits/FF128_AUDIT - + audits/bugzilla2gitlab.sh - audits/code_audit.sh
Changes:
===================================== audits/FF116_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: `9c13862f3e084cec78650fa01450f6d18aec1530` ( `FIREFOX_ESR_115_BASE` ) +- End: `ff486626d0de0e7f34d65ef000c657080ddf564d` ( `FIREFOX_116_0_3_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF117_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: ff486626d0de0e7f34d65ef000c657080ddf564d ( `FIREFOX_116_0_3_RELEASE` ) +- End: 6f3830e39c76ae6d0ab19b4f9289d434d424cbe3 ( `FIREFOX_117_0_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF118_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@6f3830e39c76ae6d0ab19b4f9289d434d424cbe3 ( `FIREFOX_117_0_RELEASE` ) +- End: tor-browser@a928b6c0612a2690852fa3b5d13efc2a80868a90 ( `FIREFOX_118_0_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF119_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@a928b6c0612a2690852fa3b5d13efc2a80868a90 ( `FIREFOX_118_0_RELEASE` ) +- End: tor-browser@7ab3cc0103090dd7bfa02e072a529b9fc784ab4e ( `FIREFOX_119_0_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF120_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@7ab3cc0103090dd7bfa02e072a529b9fc784ab4e ( `FIREFOX_119_0_RELEASE` ) +- End: tor-browser@dedee7a8c6cbabc80294733634360f6fbeeeadc0 ( `FIREFOX_120_0_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF121_AUDIT ===================================== @@ -0,0 +1,28 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@dedee7a8c6cbabc80294733634360f6fbeeeadc0 ( `FIREFOX_120_0_RELEASE` ) +- End: tor-browser@a32b8662993085139ac91212a297123b632fc1c0 ( `FIREFOX_121_0_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +#### 1add9d4c13a6493e670d01b38f4eb839c53bf1ba +- Mozilla 1815739: Support using Firefox as default PDF reader on Android +- https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43159 +- Review Result: SAFE + +#### a6562d5849a78c58340bb3d9b975f1208db4401d +- Mozilla 1852340: Implement a new "report broken site" feature for desktop Firefox +- https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43160 +- Review Result: SAFE
===================================== audits/FF122_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@a32b8662993085139ac91212a297123b632fc1c0 ( `FIREFOX_121_0_RELEASE` ) +- End: tor-browser@7e38fabb90748649da04ed45a2f80d68423362d9 ( `FIREFOX_122_0_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF123_AUDIT ===================================== @@ -0,0 +1,30 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@7e38fabb90748649da04ed45a2f80d68423362d9 ( `FIREFOX_122_0_RELEASE` ) +- End: tor-browser@f8704c84a751716bad093b9bdc482db53fe5b3ea ( `FIREFOX_123_0_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`) + +#### 14797b7fa8c5df0332ba5d422803dbcdf548c056 +#### eb73825495faf333a4fe812316ac38e138f5bf8d +#### 818788a96a700c6d44a17ab1e932de96cc45eac6 +#### c0aa048b3918e367e9fd84442695f1fbb2087f30 +- https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43161 +- Mozilla 1852900: Pass HTTPS requests to native resolver thread +- Mozilla 1852902: Allow nsINativeDNSResolverOverride to override native HTTPS records +- https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43161 +- Review Result: SAFE
===================================== audits/FF124_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@f8704c84a751716bad093b9bdc482db53fe5b3ea ( `FIREFOX_123_0_RELEASE` ) +- End: tor-browser@eb063e98ca624ff7d430a9b9aa356381f49e2e5a ( `FIREFOX_124_0_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF125_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@eb063e98ca624ff7d430a9b9aa356381f49e2e5a ( `FIREFOX_124_0_RELEASE` ) +- End: tor-browser@59577ab1445892568bafb39124e5757a307177f2 ( `FIREFOX_125_0_BUILD1` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF126_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: 59577ab1445892568bafb39124e5757a307177f2 ( `FIREFOX_125_0_BUILD1` ) +- End: 5889d9823cc5975561827262efeb24464360402c ( `FIREFOX_126_0_BUILD1` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF127_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: 5889d9823cc5975561827262efeb24464360402c ( `FIREFOX_126_0_BUILD1` ) +- End: e480e7382673f60d2f8590e7018d291b52e982b0 ( `FIREFOX_127_0b1_RELEASE` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/FF128_AUDIT ===================================== @@ -0,0 +1,20 @@ +# General + +The audit begins at the commit hash where the previous audit ended. Use code_audit.sh for creating the diff and highlighting potentially problematic code. The audit is scoped to a specific language (currently C/C++, Rust, Java/Kotlin, and Javascript). + +The output includes the entire patch where the new problematic code was introduced. Search for `XXX MATCH XXX` to find the next potential violation. + +`code_audit.sh` contains the list of known problematic APIs. New usage of these functions are documented and analyzed in this audit. + +## Firefox: https://github.com/mozilla/gecko-dev.git + +- Start: tor-browser@e480e7382673f60d2f8590e7018d291b52e982b0 ( `FIREFOX_127_0b1_RELEASE` ) +- End: tor-browser@9352d2be309c27f0e93471e2bb3352d7cfb76052 ( `FIREFOX_128_0b1_BUILD1` ) + +### Languages: +- [x] java +- [x] cpp +- [x] js +- [x] rust + +Nothing of interest (using `code_audit.sh`)
===================================== audits/bugzilla2gitlab.sh ===================================== @@ -0,0 +1,122 @@ +#!/usr/bin/env bash + +echoerr() { echo "$@" 1>&2; } + +if [ "$#" -lt 3 ]; then + echoerr "Usage: $0 firefox-version gitlab-audit-issue-number reviewers... > output.md" + exit 1 +fi + +# Check pre-conditions +check_exists() { + local cmd=$1 + if ! which ${cmd} > /dev/null ; then + echoerr "missing ${cmd} dependency" + exit 1 + fi +} + +check_exists wget +check_exists jq +check_exists sed +check_exists perl + +# assign arguments to named variables +firefox_version=$1 +audit_issue=$2 +reviewers="${@:3}" + +# check valid esr version +if ! [[ "${firefox_version}" =~ ^[1-9][0-9]{2}$ ]]; then + echoerr "invalid Firefox version (probably)" + exit 1 +fi + +# check valid issue number +if ! [[ "${audit_issue}" =~ ^[1-9][0-9]{4}$ ]]; then + echoerr "invalid gitlab audit issue number (probably)" + exit 1 +fi + +# download bug list +json=/tmp/${firefox_version}.json +bugzilla_query="https://bugzilla.mozilla.org/buglist.cgi?j_top=OR&f1=target_milestone&am..." +# you can get this from the 'REST' link at the bottom of the prevoius bugzilla query ^^; +bugzilla_json_query="https://bugzilla.mozilla.org/rest/bug?include_fields=id,summary,status&b..." + +wget "${bugzilla_json_query}" -O ${json} + +echo "### [Bugzilla Query](${bugzilla_query})" +echo "" + +issue_count=$(jq '.bugs | length' ${json}) +counter=0 +jq '.bugs | sort_by(.id)[] | "(.id)|(.summary)"' ${json} | while IFS='|' read -r id summary; do + + # indexing + counter=$((counter + 1)) + + from=$counter + through=$((counter + 499)) + if ((to > issue_count)); then + to=$issue_count + fi + + # break up into sections or else gitlab falls over + if ((counter % 500 == 1)); then + echo "<details>" + echo " <summary>Resolved Firefox ${firefox_version} Bugzilla Issues ${from} through ${through}</summary>" + echo "" + fi + + # bugzilla info + id="${id:1}" + summary="${summary:0:-1}" + [[ ${#summary} -gt 90 ]] && summary_short="${summary:0:87}..." || summary_short="${summary}" + + # we need to escape printed strings for markdown + md_escape() { + local input="$1" + # jesus I'm sorry + echo "${input}" | sed 's/[][\`*_{}<>()#+-.~]/\&/g' + } + + md_summary=$(md_escape "${summary}") + md_summary_short=$(md_escape "$summary_short") + + # we need to urlencode the strings used in the new issue link + url_encode() { + local input="$1" + echo "${input}" | perl -MURI::Escape -wlne 'print uri_escape $_' + } + + # parent issue + bugzilla_url="https://bugzilla.mozilla.org/show_bug.cgi?id=$%7Bid%7D" + # review issue title + new_issue_title=$(url_encode "Review Mozilla ${id}: ${summary_short}") + # review issue description + new_issue_description=$(url_encode "### Bugzilla: ${bugzilla_url}")%0A$(url_encode "/label ~"14.0 stable" ~FF128-esr ~Next")%0A$(url_encode "/relate tpo/applications/tor-browser-spec#${audit_issue}")%0A%0A$(url_encode "<!-- briefly describe why this issue needs further review -->")%0A + # url which create's new issue with title and description pre-populated + new_issue_url="../../../../tor-browser/-/issues/new?issue[title]=${new_issue_title}&issue[description]=${new_issue_description}" + + # em-space + em=" " + counter_string=$(printf "%04i" ${counter}) + + echo "- **${counter_string}**${em}<kbd>[Create Issue](${new_issue_url})</kbd>${em}[**${id}**: ${md_summary}](${bugzilla_url})" + + + if ((counter % 500 == 0 )) || (( counter == issue_count )); then + # checklist of engineers that have triaged this block + echo "</details>" + echo + echo "**Triaged by:**" + for reviewer in $reviewers; do + echo "- [ ] **${reviewer}**" + done + echo + elif ((counter % 25 == 0 )); then + # add a hrule every 25 to break things up visually + echo "---" + fi +done
===================================== audits/code_audit.sh ===================================== @@ -138,7 +138,7 @@ case "${SCOPE}" in initialize_rust_symbols ;; "js" ) - EXT="js jsm" + EXT="js jsm mjs" initialize_js_symbols ;; * ) @@ -172,9 +172,9 @@ rm -f "${REPORT_FILE}" # of said commit
# Flashing Color constants -export GREP_COLOR="05;37;41" +export GREP_COLORS="mt=05;37;41"
-for COMMIT in $(git rev-list --ancestry-path $OLD~..$NEW); do +for COMMIT in $(git log --format="%H" $NEW ^$OLD); do TEMP_DIFF="$(mktemp)"
echo "Diffing $COMMIT..."
View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-spec/-/compare/10...