Bang searches without an actual query (i.e. just searching "!gh") will
now redirect to the home page. I guess people do this for some reason
and don't like that it redirects to the correct bang result URL, but
without an actual search term.
Fixes#595
Recent changes to ads in search results caused Whoogle to display ads
for certain searches. In particular, ads recently started appearing
grouped into one div, as opposed to a singular ad per div. This was
accompanied by the div label "ads" (instead of just "ad"), which threw
off the existing ad filter. The ad keyword blacklist has been updated
accordingly, and has been enhanced to only check against alpha chars for
each label.
This only seems to have affected English language searches, and only for
very specific searches.
Currency amounts returned by google seem to randomly include unicode
chars ('\xa0' noted in #642) which broke the currency calculator
included in the project. This ensures that only strings that can be
converted to float are ever used in the conversion.
Fixes#642
Rather than only checking for an available update on app init, the check
for updates now performs the check once every 24 hours on the first
request sent after that period.
This also now catches the requests.exceptions.ConnectionError that is
thrown if the app is initialized without an active internet connection.
Fixes#649
Removes dependency on class names for creating the "my ip" info card in
the results list for searches pertaining to the user's public IP.
Adds test to prevent this from happening again.
Note to anyone reading this and looking to contribute: please avoid
using hardcoded class names at all costs. This approach of
creating/removing content just results in issues if/when Google decides
to introduce/remove class names from the result page.
Fixes#657
Introduces a header for switching between result types (i.e. "All", "News",
etc) that is consistent between the different result types. Previously, image
results had a tab header that was formatted in a drastically different manner,
which was jarring when switching from a different result page to the Images
page.
Created a G class enum to reference class names returned in search
results. As noted in the class doc, this should only be used/updated as
a last resort, as class names change frequently. For some instances,
such as replacing the tbm tab, it's a lot easier to just replace by
header name than attempting to replace it based on how the element is
structured.
Also updated a few styles to revert the latest styling changes being
applied by Google.
Co-authored-by: jacr13 <ramos.joao@protonmail.com>
Co-authored-by: Ben Busby <contact@benbusby.com>
Google's latest formatting changes broke the modifications made when enabling
`WHOOGLE_MINIMAL`. This updates the result filtering to work with the new
changes.
Fixes#634
min-width was previously set to 736px for all screen sizes, which forced
content off screen for smaller devices such as mobile phones. This
modifies the search stylesheet to only apply a min-width style to
devices > 800px wide.
There have been some recent formatting changes made by Google for search
results that do not look good (especially for dark themes). This
mostly overrides those styles to resemble the original Whoogle
result formatting.
Initializing the DDG bangs when running whoogle for the first time
creates an indeterminate amount of delay before the app becomes usable,
which makes usability tests (particularly w/ Docker) unreliable. This
moves the bang json init to a background thread and writes a temporary
empty dict to the bangs json file until the full bangs json can be used.
As reported in #593, the XML response body returned for search
suggestions can apparently contain invalid XML elements. This catches
the error and returns an empty suggestion list instead of erroring.
Fixes#593
The country URL param ('gl') is no longer set to 'US' by default, and is
omitted from the search entirely unless explicitly set by the user. This
change was made in an attempt to cut back on the number of captchas
experienced by certain users self-hosting who experienced a decreased
amount of captchas when this configuration setting was removed.
Fixes#558
* Integrate Farside into Whoogle
When instances are ratelimited (when a captcha is returned instead of
the user's search results) the user can now hop to a new instance via
Farside, a new backend service that redirects users to working instances
of a particular frontend. In this case, it presents a user with a
Farside link to a new Whoogle (or Searx) instance instead, so that the
user can resume their search.
For the generated Farside->Whoogle link, the generated link includes the
user's current Whoogle configuration settings as URL params, to ensure a
more seamless transition between instances. This doesn't translate to
the Farside->Searx link, but potentially could with some changes.
* Expand conversion of config<->url params
Config settings can now be translated to and from URL params using a
predetermined set of "safe" keys (i.e. config settings that easily
translate to URL params).
* Allow jumping instances via Farside when ratelimited
When instances are ratelimited (when a captcha is returned instead of
the user's search results) the user can now hop to a new instance via
Farside, a new backend service that redirects users to working instances
of a particular frontend. In this case, it presents a user with a
Farside link to a new Whoogle (or Searx) instance instead, so that the
user can resume their search.
For the generated Farside->Whoogle link, the generated link includes the
user's current Whoogle configuration settings as URL params, to ensure a
more seamless transition between instances. This doesn't translate to
the Farside->Searx link, but potentially could with some changes.
Closes#554Closes#559
This implements a method for converting between various currencies. When a user
searches "<currency A> to <currency B>" (including when prefixed by a specific
amount), they are now presented with a table for quickly converting between the
two. This makes use of the currency ratio returned as the first "card" in
currency related searches, and the table is inserted into this same card.
Previously had hardcoded POST requests for all requests that didn't use
the header template (which currently is only the image tab).
Also refactored how the Filter class works. It now requires a valid
Config model to be provided, which is then set up as a class var that
the filtering functions can use as needed, rather than setting specific
values from the config as individual values (which was confusing and
sloppy).
Fixes#561
The default CSP is only helpful for some, and can break instances for
others. Since these aren't always necessary and are occasionally set by
the user's preferred reverse proxy, it is being disabled unless
explicitly enabled by setting `WHOOGLE_CSP`.
Fixes#493
This switches the param used for the "country" config setting from "cr"
(which only filters results by the country the result is hosted in) to
"gl" (which overrides server/hosting location and produces results that
are more accurate for the user's current country).
Before this change, the country config setting was (imo) pretty useless.
Allowing a user to override an instance's hosting location with their
preferred country though is way more useful, especially for public
instances that are hosted in a different country than the user.
Closes#544
Previously the load/save/apply buttons in the config menu were hidden
below all available config options and required the user to scroll to
the bottom to save changes. This made for bad ux, since for new users,
it isn't immediately apparent that selecting a new dropdown value, for
instance, doesn't instantly save the new setting. The new layout should
make it more clear that hitting "Apply" is required to save config
changes.
This expands on the current testing suite a bit by introducing a new
workflow for testing functionality within the docker container. It runs
the same test suite as the regular "test" workflow, but also performs a
health check after running the app for 10 seconds to ensure
functionality.
The buildx workflow now waits for the docker test script to finish
successfully, rather than the regular test workflow. This will hopefully
avoid situations where new images are pushed with issues that aren't
detected in regular testing of the app.
Flask's `request.url` uses `http` as the protocol, which breaks
instances that enforce `https`, since the session redirect relies on
`request.url` for the follow-through URL.
This introduces a new method for determining the correct URL to use for
these redirects by automatically replacing the protocol with `https` if
the `HTTPS_ONLY` env var is set for that instance.
Fixes#538Fixes#545
HTTPS upgrades should be handled outside of Whoogle, since Flask often
doesn't detect the right protocol when being used behind a reverse proxy
such as Nginx.
This introduces a new approach to handling user sessions, which should
allow for users to set more reliable config settings on public instances.
Previously, when a user with cookies disabled would update their config,
this would modify the app's default config file, which would in turn
cause new users to inherit these settings when visiting the app for the
first time and cause users to inherit these settings when their current
session cookie expired (which was after 30 days by default I believe).
There was also some half-baked logic for determining on the backend
whether or not a user had cookies disabled, which lead to some issues
with out of control session file creation by Flask.
Now, when a user visits the site, their initial request is forwarded to
a session/<session id> endpoint, and during that subsequent request
their current session id is matched against the one found in the url. If
the ids match, the user has cookies enabled. If not, their original
request is modified with a 'cookies_disabled' query param that tells
Flask not to bother trying to set up a new session for that user, and
instead just use the app's fallback Fernet key for encryption and the
default config.
Since attempting to create a session for a user with cookies disabled
creates a new session file, there is now also a clean-up routine included
in the new session decorator, which will remove all sessions that don't
include a valid key in the dict. NOTE!!! This means that current user
sessions on public instances will be cleared once this update is merged
in. In the long run that's a good thing though, since this will allow session
mgmt to be a lot more reliable overall for users regardless of their cookie
preference.
Individual user sessions still use a unique Fernet key for encrypting queries,
but users with cookies disabled will use the default app key for encryption
and decryption.
Sessions are also now (semi)permanent and have a lifetime of 1 year.
Validation of the Tor connection occasionally fails with a
ConnectionError from requests, which was previously uncaught. This is
now handled appropriately (error message shown and connection dropped).
Fixes#532
Ben Busby
Nitish Yadav
DUO Labs
Joao A. Candido Ramos
සයුරි | Sayuri
Vansh Comar
Ben Busby
Ilya Prokopenko
gnuhead-chieb
Fabian Schilling