Commit Graph
274 Commits
Author SHA1 Message Date
Peter Snyder e8b0f378af add crypto to deps and build 2021-12-13 11:40:09 -06:00
Peter Snyder d9df4c8173 i made it look worse to make the robots happy 2021-12-10 15:49:58 -06:00
Peter Snyder 3c5fb368e1 move CSP code to a helper funciton, apply to stylesheet property 2021-12-10 15:06:52 -06:00
Peter Snyder 548ffac408 remove no-longer needed dep 2021-12-10 14:30:52 -06:00
Peter Snyder 90a2fb53c3 add csp protections 2021-12-10 14:15:41 -06:00
Peter Snyder ae0a93ef82 add title to speedreader docs, fixes brave/brave-browser#19182 2021-12-10 10:40:43 -06:00
mkarolin 50f7d5587b Task-related files moved from base/ to base/task/
Chromium change:

https://chromium.googlesource.com/chromium/src/+/39810e70da06baca176c0cdbc93164be1f980f59

commit 39810e70da06baca176c0cdbc93164be1f980f59
Author: Patrick Monette <pmonette@chromium.org>
Date:   Thu Oct 14 18:06:59 2021 +0000

    Reland "Move task-related files from base/ to base/task/"

    This is a reland of 092c30c5fd4def5a0c63d6f3a8953bd07768dd44

    The 2 problematic subrepos have been migrated and a couple late
    addition of the old headers were migrated.

    Original change's description:
    > Move task-related files from base/ to base/task/
    >
    > The forward headers are updated to point to the new location, and the
    > header guards were fixed.
    >
    > Bug: 1255932
2021-12-07 14:45:00 -05:00
Anton Lazarev 73895588bf remove Cargo.lock from GN inputs
Removes all `Cargo.lock` files except for the one in `build/rust`. As per [The Cargo Book](https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html):

> If you’re building a non-end product, such as a rust library that other rust packages will depend on, put `Cargo.lock` in your `.gitignore`. If you’re building an end product, which are executable like command-line tool or an application, or a system library with crate-type of `staticlib` or `cdylib`, check `Cargo.lock` into `git`. If you're curious about why that is, see ["Why do binaries have `Cargo.lock` in version control, but not libraries?" in the FAQ](https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries).

We build all of our Rust FFI components into the browser by tying them into a single crate under `build/rust`. We currently check each FFI's individual `Cargo.lock` into version control as well as `build/rust/Cargo.lock`. By the rules above, only `build/rust/Cargo.lock` should be checked in; we should add the others to `.gitignore` and remove them from the repo.
2021-11-17 15:10:43 -08:00
Anton Lazarev 3654dc3d2a exclude all but the top-level Cargo.lock from version control 2021-11-17 10:14:00 -08:00
Simon Hong 9e120103d8 Cleanup: deleted brave_vpn/url_constants.h
brave_vpn/brave_vpn_constants.h is used for all constants.
2021-11-15 21:24:35 +09:00
renovate[bot]andRenovate Bot 60c9b8c0c4 Update Rust crate time to 0.3.5 (#11076)
Co-authored-by: Renovate Bot <bot@renovateapp.com>
2021-11-13 12:13:56 -05:00
renovate[bot]andRenovate Bot fe293a26ac Update Rust crate serde_json to 1.0.70 (#11075)
Co-authored-by: Renovate Bot <bot@renovateapp.com>
2021-11-13 12:00:40 -05:00
renovate[bot]andRenovate Bot 878bbc2e6e Update Rust crate libc to 0.2.107 (#11074)
Co-authored-by: Renovate Bot <bot@renovateapp.com>
2021-11-13 11:53:47 -05:00
Ralph Giles 396d931664 speedreader: replace chrono dep with time-rs.
The maintainers of the `chrono` crate haven't been responsive to
CVE fixes. Replace our use of that crate with similar support
from the `time` crate.

This addresses `cargo audit` reports

- https://rustsec.org/advisories/RUSTSEC-2020-0159
- https://rustsec.org/advisories/RUSTSEC-2020-0071

See https://github.com/chronotope/chrono/pull/578 for discussion.
2021-11-09 14:15:22 -08:00
Ralph Giles 497bb5c817 speedreader: update rust dependencies.
Move to newer crate versions to address `cargo audit` issues.

Addresses:

- crossbeam-deque 0.8.0 Data race
  https://rustsec.org/advisories/RUSTSEC-2021-0093
- crossbeam-epoch 0.9.3 yanked
- hyper 0.14.4 request smuggling
  https://rustsec.org/advisories/RUSTSEC-2021-0078
- hyper 0.14.4 `Transfer-Encoding` data loss
  https://rustsec.org/advisories/RUSTSEC-2021-0079
- tokio 1.4.0 Task dropped in wrong thread when aborting `LocalSet` task
  https://rustsec.org/advisories/RUSTSEC-2021-0072

Unfortunately chrono hasn't been updated to address RUSTSEC-2020-0159.

Resolves brave/brave-browser#19330
2021-11-09 14:14:47 -08:00
Renovate Bot 6d599ca9bd Update Rust crate serde_json to 1.0.69 2021-11-05 23:13:57 +00:00
Renovate Bot 5b4dc8f603 Update Rust crate thiserror to 1.0.30 2021-11-05 22:52:17 +00:00
Anton Lazarev 806f000286 Merge pull request #10916 from brave/renovate/reqwest-0.x
Update Rust crate reqwest to 0.11.6
2021-11-05 13:15:23 -07:00
Renovate Bot 6fe3afe707 Update Rust crate reqwest to 0.11.6 2021-11-05 20:14:13 +00:00
Anton Lazarev 21da5af932 Merge pull request #10915 from brave/renovate/libc-0.x
Update Rust crate libc to 0.2.106
2021-11-05 13:10:31 -07:00
Renovate Bot 0f1715183c Update Rust crate libc to 0.2.106 2021-11-05 20:01:22 +00:00
Renovate Bot 6cf188308a Update Rust crate flate2 to 1.0.22 2021-11-05 18:02:56 +00:00
Ralph Giles 9f08fc28d6 speedreader: Clean up classifier::convert_map.
Replace the unrolled mapping between `HashMap` keys returned by
the feature extractor and the model input vector with a loop,
which is easier to maintain.
2021-11-02 08:14:04 -07:00
Ralph Giles 1a4c3676b6 speedreader: Avoid to_string in FeaturisingTreeSink.
Prefer borrows to explicit `clone` calls, and compare element names
against the interned string atoms instead of passing them through
`to_string` which here uses the default implementation via the
`Display` trait, which uses format calls and therefore can be
quite slow.

Unfortunately I don't think much can be done about the HashMap
keys, since `entry()` takes and owned key by value so it can
support insertion.
2021-11-02 08:14:04 -07:00
Ralph Giles 747a540c33 speedreader: Clean up documentation.
Remove obsolete references to an explicit list of readable sites,
and clarify some links and instructions.
2021-11-02 08:13:49 -07:00
Kevin Kuehler c0dd1cd60e Undo CSS parser build error 2021-09-30 16:14:34 -07:00
Kevin Kuehler 7341abd7fc Review fixups
* Remove Andrius' custom cssparser fork
  * Remove backend_ variable since we can pass RewriterType::Readability
    directly to the function now.
  * Remove dead includes
2021-09-30 15:38:53 -07:00
Kevin Kuehler ac5f13efd1 review fixups 2021-09-30 15:38:53 -07:00
Kevin Kuehler d08513dd50 remove speedreader whitelist cli flag 2021-09-30 15:38:53 -07:00
Kevin Kuehler 2c9ab5a438 speedreader: Remove IsWhitelistedForTest() API
Also rename the speedreader_rewriter->IsWhitelisted() to
speedreader_rewriter->URLLooksReadable() to avoid confusion.
2021-09-30 15:38:53 -07:00
Kevin Kuehler e7f25a7f70 remove serde dependency since only the whitelist needed it 2021-09-30 15:38:53 -07:00
Kevin Kuehler 46a07ba1b1 delete benchmarks, tests, extra files referencing rewriter rules or the
whitelist
2021-09-30 15:38:53 -07:00
Anton Lazarev 903fe473c3 remove legacy speedreader backend and adblock dependency 2021-09-30 15:38:51 -07:00
Kevin Kuehler 488235f749 speedreader: Throw out srcset if they don't look like images
Resolves https://github.com/brave/brave-browser/issues/18280
2021-09-23 15:47:44 -07:00
Kevin Kuehler ce90b1e434 speedreader: Pass result correctly for CR94
Prior to CR94 we were able to assume that the URLLoaderThrottle would
outlive the URLLoader. This is no longer the case, so we pass the
result_delegate_ to the loader.

Resolves https://github.com/brave/brave-browser/issues/18151
2021-09-15 23:45:10 -07:00
Kevin Kuehler 41d7e66eaf speedreader: HTML decode byline
Thanks to Stephen Donner for reporting this.

Resolves https://github.com/brave/brave-browser/issues/18006
2021-09-10 13:45:14 -07:00
Kevin Kuehler 347a4a9aa8 Add another constraint to sentence boundary checks
Any boundary we consider must be the end of the string or followed by a
capitalized token.
2021-09-09 23:13:31 -07:00
Kevin Kuehler 6b8bd8f43b Include abbreviation list source 2021-09-09 23:13:31 -07:00
Kevin Kuehler 098bc3c41a speedreader: Improve description truncation
For sentences that are more than 200 characters (roughly 3 sentences)
truncate the result at the first sentence. Right now some descriptions
are getting prematurely truncated on abbreviations.

Resolves https://github.com/brave/brave-browser/issues/17983
2021-09-09 23:13:31 -07:00
Kevin Kuehler c5677c6be5 speedreader tab helper and test fixes
* Use NavigationEntryCommitted() instead of DidNavigationFinish(). We
    have no guarantees about the next DidNavigationStart() being called
    first.
  * Fix throttle unit test
  * Make several review requests to SpeedreaderBrowserTest
2021-09-03 10:20:13 -07:00
Kevin Kuehler d95845e817 speedreader: Add session logic tests
We add two browser test cases for Speedreader. In these tests
Speedreader is enabled.
  (1) Run a readable page through Speedreader. Close the browser and
      open it back up. The page should restore as readable.
  (2) Navigate to a non-readable page, then to a readable page, then do
      a back navigation. The readable state should not "stick".
2021-09-02 11:37:27 -07:00
Kevin Kuehler 47f68c7677 Update entry cache in DidFinishNavigation()
Previously we were persisting the SpeedreaderExtendedInfoHandler data to
the NavigationEntry in OnDistillComplete(). Since OnDistillComplete() is
called by the throttle, the navigation entry might not be persisted yet,
so the behavior is not defined. On backward navigations we would
sometimes persist to the previous navigation entry, causing bugs where
pages would incorrectly show up as readable. Since we wait for
DidFinishNavigation, we can safely get the last committed navgiation
entry.
2021-09-02 11:37:27 -07:00
Kevin Kuehler 5bf7523d76 speedreader: Simplify delegate lifetime weak_ptr 2021-09-02 11:37:27 -07:00
Kevin Kuehler 5e3d893bed Refactor for review
* Use the DistillState enum in
    SpeedreaderExtendedInfoHandler to de-duplicate logic.
  * Move DistillState to speedreader component
  * Move SpeedreaderExtendedInfoHandler to speedreader component
  * Pass wc_getter to SpeedreaderThrottle instead of creating the
    lifetime helper
2021-09-02 11:37:26 -07:00
Kevin Kuehler 5cfaecde97 Handle results that are loaded from cached tabs
Resolves https://github.com/brave/brave-browser/issues/17355
2021-09-02 11:37:26 -07:00
Kevin Kuehler c3e335fa56 Bolt up speedreader component to tab_helper
Uses the webcontents to deliver the result to the correct tab helper. We
might want to follow this up by having the distiller thread return
tuple:

  struct DistillResult {
    bool success;
    std::string buffered_body;
  };

If success is ever false we can abort the loader.
2021-09-02 11:37:26 -07:00
Kevin Kuehler 7cd99fa3cd speedreader: Look both ways for noscript deletion
When a noscript wrapped image is found, look at preceding and following
siblings. We were getting duplicate images by not deleting the following
images.

Resolves https://github.com/brave/brave-browser/issues/17787
2021-09-01 09:48:35 -07:00
Kevin Kuehler 37effe6a04 speedreader: statistics: Fix broken test
In 7dde8faf69 the paragraph threshold was lowered to 140 from 280,
which caused this test to break. Fix the expected calculations.
2021-09-01 09:48:31 -07:00
mkarolin 4608286bf3 IWYU: early_hints.mojom.h 2021-08-18 10:57:05 -04:00
Anton Lazarev 5664b626c5 update cargo lockfiles after renovate changes 2021-07-20 12:38:41 -07:00