This allows the user to enable their preferred settings in a variety of
ways, depending on their deployment preference. Values added to
whoogle.env can be enabled using WHOOGLE_DOTENV=1, in which case all
values in the env var file will overwrite defaults or user provided
settings.
Co-authored-by: Ben Busby <benbusby@protonmail.com>
Eventually this should be part of a separate mypy ci build, but right
now it's just a general guideline. Future commits and PRs should be
validated for static typing wherever possible.
For reference, the testing commands used for this commit were:
mypy --ignore-missing-imports --pretty --disallow-untyped-calls app/
mypy --ignore-missing-imports --pretty --disallow-untyped-calls test/
* Add custom CSS field to config
This allows users to set/customize an instance's theme and appearance to
their liking. The config CSS field is prepopulated with all default CSS
variable values to allow quick editing.
Note that this can be somewhat of a "footgun" if someone updates the
CSS to hide all fields/search/etc. Should probably add some sort of
bandaid "admin" feature for public instances to employ until the whole
cookie/session issue is investigated further.
* Symlink all app static files to test dir
* Refactor app/misc/*.json -> app/static/settings/*.json
The country/language json files are used for user config settings, so
the "misc" name didn't really make sense. Also moved these to the static
folder to make testing easier.
* Fix light theme variables in dark theme css
* Minor style tweaking
The app/utils/*_utils weren't named very well, and all have been updated
to have more accurate names.
Function and class documention for the utils have been updated as well,
as part of the effort to improve overall documentation for the project.
Introduces a new content security policy header for responses to all
requests to reduce the possibility of ip leaks to outside connections.
By default blocks all inline scripts, and only allows content loaded
from Whoogle.
Refactors a few small inline scripting cases in the project to their own
individual scripts.
Requiring authentication for accessing the opensearch template prevents
the browser from accessing the file when adding as a default search
engine. This removes the authentication requirement from the opensearch
route, which should never provide any sensitive information anyways.
Bang operator can now be placed anywhere in the query, to allow for peak
efficiency in stream of consciousness querying (i.e. `big !reddit
chungus` will search reddit for big chungus`).
Fixes#196
* Adds the ability to redirect reddit.com to libredd.it using the existing
"site alts" config setting.
This adds the WHOOGLE_ALT_RD environment variable for optionally
redirecting reddit links to libreddit
(https://github.com/spikecodes/libreddit).
* Include libreddit in home page site alt note
Pip installs of whoogle search were missing access to the misc/ folder,
which previously contained the language and country json files. These
have been moved to app/misc, and the previous root level misc/ was
renamed to config/ (since it now only contains the tor config files).
Bump to 0.3.1.
Heroku instances were using the base http url when formatting the
opensearch.xml template. This adds a new routing utility, "needs_https",
which can be used for determining if the url in question needs
upgrading.
With javascript disabled, searches could not be submitted on the results
page using the "Enter" key. Adding a hidden submit button to the header
template resolves this issue.
The lxml dependency in the project was fairly unnecessary, and made the
initial build time for the project considerably slower. This replaces
all instances of lxml with either the default html parser (for bs4
constructors) or the built in xml.etree package (for search suggestion
parsing).
Introduces a new javascript "utils" file, which includes a check for
matching the query against a set of tracking number regexes on page
load. If a match is found, the script prepends a link to the
(presumably) appropriate tracking page.
Referenced in #98
Moves the language and country dicts from the config model to json files
that are loaded during app init and stored in the app config dict. This
substantially improves the readability of the config model and allows
for much more sensible loading of the language/country options.
This adds a step in the filter process to wrap the "people also ask"
section in a <details> element, which automatically collapses the
contents of the section. Clicking/tapping the details element expands
the view as normal.
See #113
The BeautifulSoup constructur in gen_nojs needed to explicitly set
features='lxml' to silence a warning from the library.
Also temporarily disabled the site alts test since the results are too
unreliable. This should be moved to a unit test instead.
The pagination footer on the results page after page 2 has three actions
(beginning, next, previous). The footer filter was updated to remove
items with more than three actions to fix this.
See #131
* Add ability to configure site alts w/ env vars
Site alternatives (i.e. twitter.com -> nitter.net) can now be configured
using environment variables:
WHOOGLE_ALT_TW='nitter.net' # twitter alt
WHOOGLE_ALT_YT='invidio.us' # youtube alt
WHOOGLE_ALT_IG='bibliogram.art/u' # instagram alt
Updated testing to confirm results have been modified.
* Add site alt vars to docker settings and readme
Heroku app instances have been notoriously bad at having the instance
automatically upgraded to https. This adds a step in the before request
decorator to always upgrade heroku apps, since they're always deployed
with the certificate, but never configured to upgrade automatically.
Fixes#153
The invidious instance has been updated to invidious.snopyta.org, since
this instance is more reliable and has more users according to
instances.invidio.us
All site alternative redirects now redirect without the 'www' subdomain,
since most of the alternative sites don't have this subdomain set up.
The resolution for enabling full support for search + suggestions in
Chrome is to remove the "method" tag altogether for any Chrome based
browser. Any inclusion of this tag seems to break the search suggestion
feature, and makes the user add the search engine manually.
Dark mode, country, interface language, and search language configs
can now be set in the search query by appending each option as a
url parameter.
Supported args are: 'dark', 'lang_search', 'lang_interface', and 'ctry'
Ex: /search?q=%s&dark=1&lang_search=lang_en...
These config settings persist across page navigation and switching
result type, but will be reset if the main search bar is used.
See #144
The body tag of the home page was previously hidden until the page was
finished loading to prevent a flash of unstyled content, but this broke
functionality for users who disallow javascript. This adds in a new
noscript tag to manually enable visibility of the body element, as well
as automatically displaying the config section (since its visibility is
also typically handled by javascript).
* Use relative links instead of absolute
This allows for hosting under a subpath. For example if you want to host
whoogle at example.com/whoogle, it should work better with a reverse proxy.
* Use relative link for opensearch.xml
Occasionally, Firefox will send the search suggestion
string to the server without a mimetype, resulting in the suggestion
only appearing in Flask's `request.data` field. This field is typically
not used for parsing arguments, as the documentation states:
Contains the incoming request data as string in case it came with a
mimetype Flask does not handle.
This fix captures the bytes object sent to the server and parses it into
a normal query to be used in forming suggestions.
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
The header template was using Google's classes for the "Whoogle" logo,
which meant keeping up with their list of colors used in the logo. The
template was updated to only ever use the Whoogle logo color.
Accordingly, the logo specific styling in filter.py was removed, since
it is no longer needed.
Also removes all links to the shopping tab, as it seems that the
majority of the links to items are Google specific links (usually
google.com/aclk links without any discernible param for determining the
true location for the link). The shopping page should be addressed
separately with unique filtering/formatting. Further tracking of this
task will be followed in #136.
Initialization of the app now includes generation of a ddg-bang json
file, which is used for all bang style searches afterwards.
Also added search suggestion handling for bang json lookup. Queries
beginning with "!" now reference the bang json file to pull all keys
that match.
Updated test suite to include basic tests for bang functionality.
Updated gitignore to exclude bang subdir.
The javascript controller has been updated to include a call to focus
the cursor on the search field. This previously had only been seen on
Firefox, and was assumed to be a weird FF-specific bug. Adding in a
timeout to allow elements to finish loading allows the field to be
focused as expected.
Also updated the README to include clarification for IP address
tracking.
Improves clarity of the meaning behind the "Country" filter -- Google
seemingly uses this value to only return results that are hosted in a
particular country, as evidenced in the search differences highlighted
in #123. It now mentions that the results are filtered by website
hosting location.
Also, now that invidio.us is shut down, the fallback URL (invidiou.site)
is now used instead.
* Implemented new dark theme
Now uses a dedicated css file for all dark theme color changes, rather
than replacing color codes directly.
Color theme is from discussion in #60.
* Minor link color update
Reconfigured template to only use method parameter if set to search via
POST request (which is the default).
Apparently Chrome/Chromium based browsers don't like non-GET request
searches, and specifying a method caused Chrome to reject the template
altogether.
Arrow key navigation through search suggestions now populates the input
field with text content from the active selection. Navigating "down"
past the end of the suggestions list returns the active cursor to position 0,
while navigating "up" before the list of suggestions restores the
original search query and removes the active highlight from element 0.
Full implementation of social media alt redirects (twitter/youtube/instagram -> nitter/invidious/bibliogram) depending on configuration.
Verbatim search and option to ignore search autocorrect are now supported as well.
Also cleaned up the javascript side of whoogle config so that it now
uses arrays of available fields for parsing config values instead of manually assigning each
one to a variable.
This doesn't include support for Google Maps -> Open Street Maps, that
seems a bit more involved than the social media redirects were, so it
should likely be a separate effort.
Adding support to choose separately the language of search and the one for the interface (allowing a default givent by google).
Co-authored-by: Joao <ramos.joao@protonmail.com>
This is a proof of concept! The code works, but uses hardcoded operators
and may be placed in the wrong file/class.
The best-case scenario would be the possibility to use the 13.000+ ddg
operators, but I don't know if that's possible without having to
redirect to duckduckgo first.
* Project refactor (#85)
* Major refactor of requests and session management
- Switches from pycurl to requests library
- Allows for less janky decoding, especially with non-latin character
sets
- Adds session level management of user configs
- Allows for each session to set its own config -- users with blocked cookies fall back to the "default" profile (same usage as before)
- Updates key gen/regen to more aggressively swap out keys after each
request
* Added ability to save/load configs by name
- New PUT method for config allows changing config with specified name
- New methods in js controller to handle loading/saving of configs
* Result formatting and removal of unused elements
- Fixed question section formatting from results page (added appropriate
padding and made questions styled as italic)
- Removed user agent display from main config settings
* Minor change to save config button label (now "Save As...")
* Fixed issue with "de-pickling" of flask session
Having a gitignore-everything ("*") file within a flask session folder seems to cause a
weird bug where the state of the app becomes unusable from continuously
trying to prune files listed in the gitignore (and it can't prune '*').
* Switched to pickling saved configs
* Updated ad/sponsored content filter and conf naming
Configs are now named with a .conf extension to allow for easier manual
cleanup/modification of named config files
Sponsored content now removed by basic string matching of span content
* Version bump to 0.2.0
* Fixed request.send return style
* Moved custom conf files to their own directory
* Refactored whoogle session mgmt
Now allows a fallback "default" session to be used if a user's browser
is blocking cookies
* Reworked pytest client fixture to support new session mgmt
* Added better multilingual support, updated filter
Results page now includes method for switching to "All Languages" from
whichever language is specified as the primary in the config (see #74).
Also removes the non-Whoogle links from the page footer, leaving only
the page navigation controls
Added support for the date range filter on the results page, though I'd
still recommend using the ":past <unit>" query instead.
* Removed no-cache enforcement, minor styling/formatting improvements
* Improving ad filtering for non-English languages
* Added footer to results page
Added enter key submit on results page
Added results type carryover for subsequent searches on results page
Removed redundant header on image search results
Basic autocomplete/search suggestion functionality added
* Adds new GET and POST routes for '/autocomplete' that accept a string query and returns an array of suggestions
* Adds new autoscript.js file for handling queries on the main page and results view
* Updated requests class to include autocomplete method
* Updated opensearch template to handle search suggestions
* Added header template to allow for autocomplete on results view
* Updated readme to mention autocomplete feature
* Added country and safe search config options
* Updated handling of parser error in results test
* Improved handling of default country
* Added 1px empty gif fallback as a replacement for images that fail to load
* Putting '! ' at the beginning of the query now redirects to the first search result
Signed-off-by: Paul Rothrock <paul@movetoiceland.com>
* Moved get_first_url outside of filter class
Signed-off-by: Paul Rothrock <paul@movetoiceland.com>
See https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections
301 redirections do not keep the request method intact, and can occasionally be changed from POST to GET
308 redirections always keep the request method, which is necessary for all POST search requests
* Adding HTTPS enforcement
Command line runs of Whoogle Search through pip/pipx/etc will need the
`--https-only` flag appended to the run command.
Docker runs require the `use_https` build arg applied.
* Update README.md
Moved https-only note to top of docker run command, updated pip runner help output
* Dockerfile: removed HTTPS enforcement, updated PORT setting
Dockerfile no longer enforces an HTTPS connection, but still allows for
setting via a build arg. The Flask server port is now configurable as a
build arg as well, by setting a port number to "whoogle_port"
* Fixed incorrect port assignment
This addresses #18, which brought up the issue of searching with Whoogle
with the search instance set to always use a specific container in
Firefox Container Tabs.
Could also be useful if you want to share your search results or
something, I guess. Though nobody likes when people do that.
* Added language configuration support
Main page now has a dropdown for selecting preferred language of
results.
Refactored config to be its own model with language constants.
* Added more language support
Interface language is now updated using the "hl" arg
Fixed chinese traditional and simplified values
Updated decoding of characters to gb2312
* Updated to use conditional decoding dependent on language
* Updated filter to not rely on valid config to work properly
* Ignore venv when building docker file
* Remove reference to 8888 port
It wasn't really used anywhere, and setting it to 5000 everywhere removes ambiguity, and makes things easier to track and reason about
* Use waitress rather than Flask's built in web server
It's not production grade
* Actually add waitress to requirements
Woops!
Pushing straight to master since this is an extremely simple fix, with
a pretty large performance benefit.
The Phyme library used for generating a User Agent rhyme was consuming
an absolute unit of memory. Now that it's removed, it's using about 10x
less memory, at the cost of User Agents being not as funny anymore.
Config options now allow setting a "root url", which defaults to the
request url root. Saving a new url in this field will allow for proper
redirects and usage of the opensearch element.
Also provides a possible solution for #17, where the default flask redirect method redirects to
http instead of https.
Now implemented as a flask global variable reads from the same json file
as before, but doesn't crash if it does not find an existing file.
Removed user config creation from run script
Added <meta name="referrer" content="no-referrer"> to all whoogle
templates
Refactored search route to use conditionally use either request.args or
request.form, depending on rest call (get vs post respectively)
Switched encoding from utf-8 to unicode-escape in an effort to support multiple
languages besides English.
Updated image results page formatting to fix bad image links (added TODO
for adding full res image link for each image result).
Updated README to include libcurl and libssl install instructions for
manual setup.
The implementation of POST search support comes with a few benefits. The
most apparent is the avoidance of search queries appearing in web server
logs -- instead of the prior GET approach (i.e.
/search?q=my+search+query), using POST requests with the query stored in
the request body creates logs that simply appear as "/search".
Since a lot of relative links are generated in the results page, I came
up with a way to generate a unique key at run time that is used to
encrypt any query strings before sending to the user. This benefits both
regular text queries as well as fetching of image links and means that
web logs will only show an encrypted string where a link or query
string might slip through.
Unfortunately, GET search requests still need to be supported, as it
doesn't seem that Firefox (on iOS) supports loading search engines by
their opensearch.xml file, but instead relies on manual entry of a
search query string. Once this is updated, I'll probably remove GET
request search support.
Images were previously directly fetched from google search results,
which was a potential privacy hazard. All image sources are now modified
to be passed through shoogle's routing first, which will then fetch raw
image data and pass it through to the user.
Filter class was refactored to split the primary clean method into
smaller, more manageable submethods.
The image results page seems to have different formatting from non-image
results pages. Should probably revisit this at some point and try to
style the image results page to be more in line with other result types.
- Updated Dockerfile to include chmod of run script
- Added app.json for Heroku quick deploy
- Removed unused function var in js controller
- Moved requirements back to root of repo
- Added Codebeat report to readme
Curl requests and user agent related functionality was moved to its own
request class.
Routes was refactored to only include strictly routing related
functionality.
Filter class was cleaned up (had routing/request related logic in here,
which didn't make sense)
Shimul
Ben Busby
Tomasz Borychowski
Roman Štefko
Ziga Zajc
Ben Busby
bugbounce
curlpipe
Marvin Borner
Joao A. Candido Ramos
Paul Rothrock
Jake Howard
Fabian Schliski
Ben Busby
Ben Busby