Adds a new advanced search icon alongside the result tabs for switching
to a different country from the result page.
This will obviously get populated with other methods of filtering
results, but for now it's just the country selector.
Session validation includes a method for determining the proxy host url,
but previously did not include the path for the initial request. This
caused a situation where users with a new session would not be able to
complete their first search, since the session validation follow-through
url did not include the actual path for their search query.
The method now includes a flag for only extracting the root url, which
is needed for creating full urls in the content filter.
Fixes#708
This adds a new "temporary" config section of the results view, where a
user can now change the country that their results come from without
changing their default config settings.
Closes#322
Queries performed in a different language than what is configured
contain a result div that prompts the user to configure their language
preferences using google's preferences page.
Since we want all language configuration to occur on Whoogle only, we
can safely remove this result div.
Fixes#444Fixes#386
Farside can now redirect quora links to querte instances and imdb links
to libremdb instances. This updates Whoogle to perform link replacements
for both services when site alts are configured.
For users running local instances of service alternatives such as
invidious, the alt replacement procedure broke if the scheme of the
original service (almost always https) didn't match the scheme of their
defined local service (likely http).
This adds a small check to see if the alt has a defined scheme, and if
so, removes the original scheme for that result.
Fixes#806
Wikipedia -> Wikiless redirects always result in an english language
result, even if the Wikipedia result would've been in a non-english
language. This is due to Wikipedia using language specific subdomains
(i.e. de.wikipedia.org, en.wikipedia.org, etc) whereas Wikiless uses a
"lang" url param.
This has been fixed by inspecting the subdomain of the wikipedia link
and passing that value to Wikiless as the lang param if it's determined
to be a language specific value (currently just looking for a 2-char
subdomain).
See #805
If Whoogle is accessed on a non-standard port _and_ proxied,
this port is lost to the application and `element['src']`s are
incorrectly formed (omitting port).
HTTP x-Forwarded-Host will contain this front port number in
a typical Nginx reverse proxy configuration.
The majority of image links and links that are not handle by whoogle are not
opening in new tabs, this allow links that are not related to the application
to open in new tabs.
Due to how instances installed with pip seem to have issues storing
unrelated files in the same directory as sessions, exception handling
during session validation has been expanded to blindly ignore all
exceptions. This portion of the code is more for maintainers of large
public instances with a bunch of users who block cookies anyways, so
having basic app functionality break down as a result shouldn't be the
default.
Country config value should be checked against the valid value when
updating the home page config, not the other way around. This can lead
to a state where a user sets up an invalid country value, but can still
be matched against a correct value that is part of the invalid value
(i.e. "countryUK" is invalid, but would match against the correct value,
"UK")
Also minor refactor of where the session file size validation occurs.
For pip installed instances of Whoogle, there seems to be an issue where
files other than sessions are being stored in the same directory as the
sessions. From a brief investigation, this does not seem to be caused by
Whoogle, since Flask-Session objects are the only files stored in that
directory. It could be an issue with the library that is being used for
sessions, however.
Regardless, the app shouldn't crash when trying to validate and remove
invalid sessions, so a file size limit of 4KB was imposed during
validation. Any file found in the session directory that exceeds this
size limit will be ignored.
Fixes#777Fixes#793
Added password authentication for tor control port.
For user configuration of access to tor control port. This file should be
heavily restricted in file system.
Co-authored-by: MadcowOG <madcowog@Arch-Main.localdomain>
Similar issue to #629, but the result page uses a different script for
handling user input, so the fix was not applied appropriately.
It has been fixed for this view now.
Google updated their styling of the result page, which broke some
components of Whoogle's result page styling (namely the result div
backgrounds for dark mode).
The GClasses class has been updated to keep track of what class names
have been updated to, and roll them back to a value that works for
Whoogle. A function was added that loops through new class names and
replaces them with their older counterparts.
The previous implementation of autocomplete/suggestions on the front end
resulted in a situation where input and keydown events were constantly
being added to the search input bar. This was refactored to set up the
events only once and process suggestion navigation and appending
suggestions separately with different functions.
This has been tested on both an Android simulator, as well as an Android
tablet and seems to work as expected.
Fixes#370Fixes#629
This seems to be caused by an odd behavior related to Flask sessions and
instances of Whoogle installed via pip. I didn't investigate it too
much, since catching and ignoring the result doesn't impact Whoogle
functionality at all (configuration and session values persist as
normal). Since this doesn't affect non-pip instances, I don't believe it
to be a fault within Whoogle itself.
Fixes#765
A user reported a bug where searches with a leading slash (in this case:
"/e/OS apps" were interpreted as a Google specific link when clicking
the next page of results.
This was due to the behavior that Google's search results exhibit, where
internal links for pages like support.google.com are delivered with
params like "?q=/support" rather than a direct link. This fixes that
scenario by checking the "q" param value against the user's original
query to ensure they don't match before assuming that the result is
intended as a redirect.
Fixes#776
It appears that result links beginning with '/url' were mistakenly
commited with an inefficient filtering process in its place. With the
way the code is structured, this less effective '/url' link filter took
precedence over the previous link filter, and also caused users with the
"open link in new tab" config enabled to no longer have access to that
feature.
Fixes#769
The leading slash was previously removed without noticing it was part of a
string replacement in #734. This caused the href of "View Image" contain a
leading "/" which is wrong.
Pages in the Whoogle footer that by default route to Google pages were
previously being removed, but caused results that also routed to similar
pages to no longer be accessible. This was due to the removal of the
'/url' endpoint that Google uses for each result.
To fix this, the result link is now parsed so that the domain of the
result can be checked against the disallowed G page list. Since results
are delivered in a "/url?q=<domain>" format -- even for pages to
Google's own products -- and the footer links are formatted as
"<product>.google.com", footer links are removed and result links are
parsed correctly.
Fixes#747
SESSION_COOKIE_SAMESITE must be set to 'lax' to allow the user's
previous session to persist when accessing the instance from an external
link. Setting this value to 'strict' causes Whoogle to revalidate a new
session, and fail, resulting in cookies being disabled.
This could be re-evaluated if Whoogle ever switches to client side
configuration instead.
Fixes#749
The `/url` endpoint was previously used as a way of mirroring the
`/url?q=<result domain>` formatting of locations in search results from
Google. Rather than have this unnecessary intermediary step, the result
path was extracted and used as the immediate path for each result item
instead.
This endpoint hasn't been in use for many versions and has been in need
of removal for quite some time.
If a trailing slash is defined here, it causes the Whoogle instance to
redirect these element requests back to the home page, causing unwanted
behavior.
Previously, empty bang searches would redirect to the Whoogle instance
home page. This now redirects to the specific site for the bang search
instead (i.e. "!yt" without a query redirects to "youtube.com", "!gh" to
"github.com", etc)
Fixes#719
The "anon-view" translation key is the correct one to use for accessing
anonymous view within the search results. "config-anon-view" is only for
the configuration menu on the home page.
* Relativization of search results
* Fix JavaScript error when opening images
* Replace single-letter logo and remove sign-in link
* Add `WHOOGLE_URL_PREFIX` env var to support relative path redirection
The `WHOOGLE_URL_PREFIX` var can now be set to fix internal app
redirects, such as the `/session` redirect performed on the first visit
to the Whoogle home page.
Co-authored-by: Ben Busby <contact@benbusby.com>
In some rare instances (a race condition perhaps?) a
`cryptography.fernet.InvalidToken` exception is thrown resulting in
a broken connection.
This change gracefully returns a 401 error instead.
* Expand `/window` endpoint to behave like a proxy
The `/window` endpoint was previously used as a type of proxy, but only
for removing Javascript from the result page. This expands the existing
functionality to allow users to proxy search result pages (with or without
Javascript) through their Whoogle instance.
* Implement filtering of remote content from css
* Condense NoJS feature into Anonymous View
Enabling NoJS now removes Javascript from the Anonymous View, rather
than creating a separate option.
* Exclude 'data:' urls from filter, add translations
The 'data:' url must be allowed in results to view certain elements on
the page, such as stars for review based results.
Add translations for the remaining languages.
* Add cssutils to requirements
If the alt for a particular service is blank, the original source is
used instead.
Example:
1. Site alts enabled in config
2. User wants wikipedia links, not wikiless
3. WHOOGLE_ALT_WIKI set to ""
4. All available alt links redirected to farside, except wikipedia
Fixes#704
Old iOS 12 devices will pass the Referer HTTP header to the site user clicks.
Websites will know those traffic come from Whoogle search.
Adding "nofollow noopener noreferrer" solves the issue.
The time filter (past day/hour/month/etc) was using the result element
background color instead of the page background color, which wasn't
providing enough contrast with the default text color.
Wikipedia, imgur, and translate alternatives were all still using
hardcoded URLs when replaced with their respective alternative frontend.
This updates them to use farside instead.
Links that were directed at G domains were previously removed
universally, when really they only needed to be removed from the footer
to reduce possible confusion caused by mixed Whoogle and G links.
Fixes#656
Bang searches without an actual query (i.e. just searching "!gh") will
now redirect to the home page. I guess people do this for some reason
and don't like that it redirects to the correct bang result URL, but
without an actual search term.
Fixes#595
Recent changes to ads in search results caused Whoogle to display ads
for certain searches. In particular, ads recently started appearing
grouped into one div, as opposed to a singular ad per div. This was
accompanied by the div label "ads" (instead of just "ad"), which threw
off the existing ad filter. The ad keyword blacklist has been updated
accordingly, and has been enhanced to only check against alpha chars for
each label.
This only seems to have affected English language searches, and only for
very specific searches.
Currency amounts returned by google seem to randomly include unicode
chars ('\xa0' noted in #642) which broke the currency calculator
included in the project. This ensures that only strings that can be
converted to float are ever used in the conversion.
Fixes#642
Rather than only checking for an available update on app init, the check
for updates now performs the check once every 24 hours on the first
request sent after that period.
This also now catches the requests.exceptions.ConnectionError that is
thrown if the app is initialized without an active internet connection.
Fixes#649
Removes dependency on class names for creating the "my ip" info card in
the results list for searches pertaining to the user's public IP.
Adds test to prevent this from happening again.
Note to anyone reading this and looking to contribute: please avoid
using hardcoded class names at all costs. This approach of
creating/removing content just results in issues if/when Google decides
to introduce/remove class names from the result page.
Fixes#657
Introduces a header for switching between result types (i.e. "All", "News",
etc) that is consistent between the different result types. Previously, image
results had a tab header that was formatted in a drastically different manner,
which was jarring when switching from a different result page to the Images
page.
Created a G class enum to reference class names returned in search
results. As noted in the class doc, this should only be used/updated as
a last resort, as class names change frequently. For some instances,
such as replacing the tbm tab, it's a lot easier to just replace by
header name than attempting to replace it based on how the element is
structured.
Also updated a few styles to revert the latest styling changes being
applied by Google.
Co-authored-by: jacr13 <ramos.joao@protonmail.com>
Co-authored-by: Ben Busby <contact@benbusby.com>
Google's latest formatting changes broke the modifications made when enabling
`WHOOGLE_MINIMAL`. This updates the result filtering to work with the new
changes.
Fixes#634
min-width was previously set to 736px for all screen sizes, which forced
content off screen for smaller devices such as mobile phones. This
modifies the search stylesheet to only apply a min-width style to
devices > 800px wide.
There have been some recent formatting changes made by Google for search
results that do not look good (especially for dark themes). This
mostly overrides those styles to resemble the original Whoogle
result formatting.
Initializing the DDG bangs when running whoogle for the first time
creates an indeterminate amount of delay before the app becomes usable,
which makes usability tests (particularly w/ Docker) unreliable. This
moves the bang json init to a background thread and writes a temporary
empty dict to the bangs json file until the full bangs json can be used.
As reported in #593, the XML response body returned for search
suggestions can apparently contain invalid XML elements. This catches
the error and returns an empty suggestion list instead of erroring.
Fixes#593
The country URL param ('gl') is no longer set to 'US' by default, and is
omitted from the search entirely unless explicitly set by the user. This
change was made in an attempt to cut back on the number of captchas
experienced by certain users self-hosting who experienced a decreased
amount of captchas when this configuration setting was removed.
Fixes#558
* Integrate Farside into Whoogle
When instances are ratelimited (when a captcha is returned instead of
the user's search results) the user can now hop to a new instance via
Farside, a new backend service that redirects users to working instances
of a particular frontend. In this case, it presents a user with a
Farside link to a new Whoogle (or Searx) instance instead, so that the
user can resume their search.
For the generated Farside->Whoogle link, the generated link includes the
user's current Whoogle configuration settings as URL params, to ensure a
more seamless transition between instances. This doesn't translate to
the Farside->Searx link, but potentially could with some changes.
* Expand conversion of config<->url params
Config settings can now be translated to and from URL params using a
predetermined set of "safe" keys (i.e. config settings that easily
translate to URL params).
* Allow jumping instances via Farside when ratelimited
When instances are ratelimited (when a captcha is returned instead of
the user's search results) the user can now hop to a new instance via
Farside, a new backend service that redirects users to working instances
of a particular frontend. In this case, it presents a user with a
Farside link to a new Whoogle (or Searx) instance instead, so that the
user can resume their search.
For the generated Farside->Whoogle link, the generated link includes the
user's current Whoogle configuration settings as URL params, to ensure a
more seamless transition between instances. This doesn't translate to
the Farside->Searx link, but potentially could with some changes.
Closes#554Closes#559
This implements a method for converting between various currencies. When a user
searches "<currency A> to <currency B>" (including when prefixed by a specific
amount), they are now presented with a table for quickly converting between the
two. This makes use of the currency ratio returned as the first "card" in
currency related searches, and the table is inserted into this same card.
Previously had hardcoded POST requests for all requests that didn't use
the header template (which currently is only the image tab).
Also refactored how the Filter class works. It now requires a valid
Config model to be provided, which is then set up as a class var that
the filtering functions can use as needed, rather than setting specific
values from the config as individual values (which was confusing and
sloppy).
Fixes#561
The default CSP is only helpful for some, and can break instances for
others. Since these aren't always necessary and are occasionally set by
the user's preferred reverse proxy, it is being disabled unless
explicitly enabled by setting `WHOOGLE_CSP`.
Fixes#493
This switches the param used for the "country" config setting from "cr"
(which only filters results by the country the result is hosted in) to
"gl" (which overrides server/hosting location and produces results that
are more accurate for the user's current country).
Before this change, the country config setting was (imo) pretty useless.
Allowing a user to override an instance's hosting location with their
preferred country though is way more useful, especially for public
instances that are hosted in a different country than the user.
Closes#544
Previously the load/save/apply buttons in the config menu were hidden
below all available config options and required the user to scroll to
the bottom to save changes. This made for bad ux, since for new users,
it isn't immediately apparent that selecting a new dropdown value, for
instance, doesn't instantly save the new setting. The new layout should
make it more clear that hitting "Apply" is required to save config
changes.
This expands on the current testing suite a bit by introducing a new
workflow for testing functionality within the docker container. It runs
the same test suite as the regular "test" workflow, but also performs a
health check after running the app for 10 seconds to ensure
functionality.
The buildx workflow now waits for the docker test script to finish
successfully, rather than the regular test workflow. This will hopefully
avoid situations where new images are pushed with issues that aren't
detected in regular testing of the app.
Flask's `request.url` uses `http` as the protocol, which breaks
instances that enforce `https`, since the session redirect relies on
`request.url` for the follow-through URL.
This introduces a new method for determining the correct URL to use for
these redirects by automatically replacing the protocol with `https` if
the `HTTPS_ONLY` env var is set for that instance.
Fixes#538Fixes#545
HTTPS upgrades should be handled outside of Whoogle, since Flask often
doesn't detect the right protocol when being used behind a reverse proxy
such as Nginx.
This introduces a new approach to handling user sessions, which should
allow for users to set more reliable config settings on public instances.
Previously, when a user with cookies disabled would update their config,
this would modify the app's default config file, which would in turn
cause new users to inherit these settings when visiting the app for the
first time and cause users to inherit these settings when their current
session cookie expired (which was after 30 days by default I believe).
There was also some half-baked logic for determining on the backend
whether or not a user had cookies disabled, which lead to some issues
with out of control session file creation by Flask.
Now, when a user visits the site, their initial request is forwarded to
a session/<session id> endpoint, and during that subsequent request
their current session id is matched against the one found in the url. If
the ids match, the user has cookies enabled. If not, their original
request is modified with a 'cookies_disabled' query param that tells
Flask not to bother trying to set up a new session for that user, and
instead just use the app's fallback Fernet key for encryption and the
default config.
Since attempting to create a session for a user with cookies disabled
creates a new session file, there is now also a clean-up routine included
in the new session decorator, which will remove all sessions that don't
include a valid key in the dict. NOTE!!! This means that current user
sessions on public instances will be cleared once this update is merged
in. In the long run that's a good thing though, since this will allow session
mgmt to be a lot more reliable overall for users regardless of their cookie
preference.
Individual user sessions still use a unique Fernet key for encrypting queries,
but users with cookies disabled will use the default app key for encryption
and decryption.
Sessions are also now (semi)permanent and have a lifetime of 1 year.
Validation of the Tor connection occasionally fails with a
ConnectionError from requests, which was previously uncaught. This is
now handled appropriately (error message shown and connection dropped).
Fixes#532
This checks the latest released version of Whoogle against
the current app version, and shows an "update available"
message if the current version num < latest release num.
Closes#305
The config menu has gotten out of control recently, but rather than
reducing functionality, I'm just going to set a max height for the div
and allow scrolling within the menu.
Ultimately though this indicates that the app is getting a bit too
complicated (imo). Striking a balance between customization and
minimalism is less of a priority for me nowadays though, hence why I'm
willing to let it slide for now. At some point, maybe when there are
more contributors, it could be nice to refactor this in some way so that
it isn't overwhelming to new users who are looking to customize their
instance (that's just me speculating btw, I haven't actually heard from
anyone who thinks there are too many options in that menu).
Using `format` for formatting bang queries caused a KeyError for some
searches, such as !hd (HUDOC). In that example, the URL returned in the
bangs json was `http://...#{%22fulltext%22:[%22{}%22]...`, where
standard formatting would not work due to the misidentification of
"fulltext" as a formatting key.
The logic has been updated to just replace the first occurence of "{}"
in the URL returned by the bangs dict.
Fixes#513
Due to how the response is now reformed into a new bsoup object when
bolding search query terms, creating an ip card for "my ip" searches
threw an error due to how the new bsoup object was initialized for the
"my ip" card. This passes the response in as a string instead.
Fixes#504
DDG style bang searches can now have the bang (!) at the end of
the search (i.e. "bologna w!" will now redirect to wikipedia just like
"bologna !w" would)
Since the request class is loaded prior to values being read from the
user's dotenv, the WHOOGLE_RESULT_PER_PAGE var wasn't being used for
searches.
This moves the definition of the base search url to be intialized in the
request class to address this issue.
Fixes#497
variables.css doesn't need to be loaded by any template, since
WHOOGLE_CONFIG_STYLE loads those values by default when not set
explicitly. Loading the stylesheet caused the logo colors to be
persistent unless set individually.
Sorry @gripped for sneaking all of this unnecessary color in...
Fixes#492
This modifies the search result page by bold-ing all appearances
of any word in the original query. If portions of the query are in
quotes (i.e. "ice cream"), only exact matches of the sequence of
words will be made bold.
Co-authored-by: Ben Busby <noreply+git@benbusby.com>
Activating minimal mode should also remove all collapsed sections, if
any are found.
WHOOGLE_MINIMAL now documented in readme and app.json (for heroku).
I've gotten a bit bored of the current light/dark themes, so I'm
switching the default theme over to the Doppelganger theme, which is a
better template/jumping off point for users to use when creating custom
themes since it also provides examples for coloring each of the Whoogle
logo letters.
The levelup.gitconnected.com site is a Medium site that can also be
replaced with scribe.rip whenever privacy respecting site alternatives
are enabled in the config.
Also modified how link descriptions are updated when that config is
enabled (before it was missing replacements on quite a few
descriptions).
This introduces a new UI element for displaying the client IP
address when a search for "my ip" is used.
Note that this does not show the IP address seen by Google
if Whoogle is deployed remotely. It uses `request.remote_addr`
to display the client IP address in the UI, not the actual address
of the server (which is what Google sees in requests sent from
remote Whoogle instances).
scribe.rip is a privacy respecting front end for medium.com. This
feature allows medium.com results to be replaced with scribe.rip links,
and works for both regular medium.com domains as well as user specific
subdomains (i.e. user.medium.com).
[scribe.rip website](https://scribe.rip)
[scribe.rip source code](https://git.sr.ht/~edwardloveall/scribe)
Co-authored-by: Ben Busby <noreply+git@benbusby.com>
Ben Busby
Kian-Meng Ang
MadcowOG
Marcell Fülöp
Joao A. Candido Ramos
jan Anja
CAB233
invis-z
hoschi1337
xatier
Warren Spits
gdm85
Ilya Prokopenko
glitsj16
green1052
138138138
Nitish Yadav
DUO Labs
සයුරි | Sayuri
Vansh Comar
Ben Busby
gnuhead-chieb
Fabian Schilling
gripped
Yadomin