Support basic localization (#325)

* Replace hardcoded strings using translation json file

This introduces a new "translations.json" file under app/static/settings
that is loaded on app init and uses the user config value for interface
language to determine the appropriate strings to use in Whoogle-specific
elements of the UI (primarily only on the home page).

* Verify interface lang can be used for localization

Check the configured interface language against the available
localization dict before attempting to use, otherwise fall back to
english.

Also expanded language names in the languages json file.

* Add test for validating translation language keys

Also adds Spanish translation to json (the only non-English language I
can add and reasonably validate on my own).

* Validate all translations against original keyset, update readme

Readme has been updated to include basic contributing guidelines for
both code and translations.

README.md

@@ -26,10 +26,11 @@ Contents
     1. [Set Primary Search Engine](#set-whoogle-as-your-primary-search-engine)
     2. [Prevent Downtime (Heroku Only)](#prevent-downtime-heroku-only)
     3. [Manual HTTPS Enforcement](#https-enforcement)
-7. [FAQ](#faq)
+7. [Contributing](#contributing)
-8. [Public Instances](#public-instances)
+8. [FAQ](#faq)
-9. [Screenshots](#screenshots)
+9. [Public Instances](#public-instances)
-10. Mirrors (read-only)
+10. [Screenshots](#screenshots)
+11. Mirrors (read-only)
     1. [GitLab](https://gitlab.com/benbusby/whoogle-search)
     2. [Gogs](https://gogs.benbusby.com/benbusby/whoogle-search)
 
@@ -365,6 +366,56 @@ Note: You should have your own domain name and [an https certificate](https://le
 - Pip/Pipx: Add the `--https-only` flag to the end of the `whoogle-search` command
 - Default `run` script: Modify the script locally to include the `--https-only` flag at the end of the python run command
 
+## Contributing
+
+Under the hood, Whoogle is a basic Flask app with the following structure:
+
+- `app/`
+  - `routes.py`: Primary app entrypoint, contains all API routes
+  - `request.py`: Handles all outbound requests, including proxied/Tor connectivity
+  - `filter.py`: Functions and utilities used for filtering out content from upstream Google search results
+  - `utils/`
+    - `bangs.py`: All logic related to handling DDG-style "bang" queries
+    - `results.py`: Utility functions for interpreting/modifying individual search results
+    - `search.py`: Creates and handles new search queries
+    - `session.py`: Miscellaneous methods related to user sessions
+  - `templates/`
+    - `index.html`: The home page template
+    - `display.html`: The search results template
+    - `header.html`: A general "top of the page" query header for desktop and mobile
+    - `search.html`: An iframe-able search page
+    - `logo.html`: A template consisting mostly of the Whoogle logo as an SVG (separated to help keep `index.html` a bit cleaner)
+    - `opensearch.xml`: A template used for supporting [OpenSearch](https://developer.mozilla.org/en-US/docs/Web/OpenSearch).
+    - `imageresults.html`: An "exprimental" template used for supporting the "Full Size" image feature on desktop.
+  - `static/<css|js>`
+    - CSS/Javascript files, should be self-explanatory
+  - `static/settings`
+    - Key-value JSON files for establishing valid configuration values
+    
+
+If you're new to the project, the easiest way to get started would be to try fixing [an open bug report](https://github.com/benbusby/whoogle-search/issues?q=is%3Aissue+is%3Aopen+label%3Abug). If there aren't any open, or if the open ones are too stale, try taking on a [feature request](https://github.com/benbusby/whoogle-search/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement). Generally speaking, if you can write something that has any potential of breaking down in the future, you should write a test for it.
+
+The project follows the [PEP 8 Style Guide](https://www.python.org/dev/peps/pep-0008/), but is liable to change. Static typing should always be used when possible. Function documentation is greatly appreciated, and typically follows the below format:
+
+```python
+def contains(x: list, y: int) -> bool:
+    """Check a list (x) for the presence of an element (y)
+
+    Args:
+        x: The list to inspect
+        y: The int to look for
+
+    Returns:
+        bool: True if the list contains the item, otherwise False
+    """
+
+    return y in x
+``` 
+
+#### Translating
+
+Whoogle currently supports translations using [`translations.json`](https://github.com/benbusby/whoogle-search/blob/main/app/static/settings/languages.json). Language values in this file need to match the "value" of the according language in [`languages.json`](https://github.com/benbusby/whoogle-search/blob/main/app/static/settings/languages.json) (i.e. "lang_en" for English, "lang_es" for Spanish, etc). After you add a new set of translations to `translations.json`, open a PR with your changes and they will be merged in as soon as possible.
+
 ## FAQ
 **What's the difference between this and [Searx](https://github.com/asciimoo/searx)?**
 

app/__init__.py

@@ -33,6 +33,8 @@ app.config['LANGUAGES'] = json.load(open(
     os.path.join(app.config['STATIC_FOLDER'], 'settings/languages.json')))
 app.config['COUNTRIES'] = json.load(open(
     os.path.join(app.config['STATIC_FOLDER'], 'settings/countries.json')))
+app.config['TRANSLATIONS'] = json.load(open(
+    os.path.join(app.config['STATIC_FOLDER'], 'settings/translations.json')))
 app.config['CONFIG_PATH'] = os.getenv(
     'CONFIG_VOLUME',
     os.path.join(app.config['STATIC_FOLDER'], 'config'))

app/models/config.py

@@ -77,6 +77,19 @@ class Config:
 
         return key in self.safe_keys
 
+    def get_localization_lang(self):
+        """Returns the correct language to use for localization, but falls
+        back to english if not set.
+
+        Returns:
+            str -- the localization language string
+        """
+        if (self.lang_interface and
+                self.lang_interface in current_app.config['TRANSLATIONS']):
+            return self.lang_interface
+
+        return 'lang_en'
+
     def from_params(self, params) -> 'Config':
         """Modify user config with search parameters. This is primarily
         used for specifying configuration on a search-by-search basis on

app/routes.py

@@ -130,6 +130,9 @@ def index():
     return render_template('index.html',
                            languages=app.config['LANGUAGES'],
                            countries=app.config['COUNTRIES'],
+                           translation=app.config['TRANSLATIONS'][
+                               g.user_config.get_localization_lang()
+                           ],
                            logo=render_template(
                                'logo.html',
                                dark=g.user_config.dark),
@@ -235,6 +238,9 @@ def search():
         query=urlparse.unquote(query),
         search_type=search_util.search_type,
         config=g.user_config,
+        translation=app.config['TRANSLATIONS'][
+            g.user_config.get_localization_lang()
+        ],
         response=response,
         version_number=app.config['VERSION_NUMBER'],
         search_header=(render_template(

app/static/settings/countries.json

@@ -1,5 +1,5 @@
 [
-  {"name": "Default (none)", "value": ""},
+  {"name": "-------", "value": ""},
   {"name": "Afghanistan", "value": "countryAF"},
   {"name": "Albania", "value": "countryAL"},
   {"name": "Algeria", "value": "countryDZ"},

app/static/settings/languages.json

@@ -1,49 +1,49 @@
 [
-  {"name": "Default (none specified)", "value": ""},
+  {"name": "-------", "value": ""},
   {"name": "English", "value": "lang_en"},
-  {"name": "Afrikaans", "value": "lang_af"},
+  {"name": "Afrikaans (Afrikaans)", "value": "lang_af"},
-  {"name": "Arabic", "value": "lang_ar"},
+  {"name": "Arabic (عربى)", "value": "lang_ar"},
-  {"name": "Armenian", "value": "lang_hy"},
+  {"name": "Armenian (հայերեն)", "value": "lang_hy"},
-  {"name": "Belarusian", "value": "lang_be"},
+  {"name": "Belarusian (Беларуская)", "value": "lang_be"},
-  {"name": "Bulgarian", "value": "lang_bg"},
+  {"name": "Bulgarian (български)", "value": "lang_bg"},
-  {"name": "Catalan", "value": "lang_ca"},
+  {"name": "Catalan (Català)", "value": "lang_ca"},
-  {"name": "Chinese (Simplified)", "value": "lang_zh-CN"},
+  {"name": "Chinese, Simplified (简体中文)", "value": "lang_zh-CN"},
-  {"name": "Chinese (Traditional)", "value": "lang_zh-TW"},
+  {"name": "Chinese, Traditional (繁体中文)", "value": "lang_zh-TW"},
-  {"name": "Croatian", "value": "lang_hr"},
+  {"name": "Croatian (Hrvatski)", "value": "lang_hr"},
-  {"name": "Czech", "value": "lang_cs"},
+  {"name": "Czech (čeština)", "value": "lang_cs"},
-  {"name": "Danish", "value": "lang_da"},
+  {"name": "Danish (Dansk)", "value": "lang_da"},
-  {"name": "Dutch", "value": "lang_nl"},
+  {"name": "Dutch (Nederlands)", "value": "lang_nl"},
-  {"name": "Esperanto", "value": "lang_eo"},
+  {"name": "Esperanto (Esperanto)", "value": "lang_eo"},
-  {"name": "Estonian", "value": "lang_et"},
+  {"name": "Estonian (Eestlane)", "value": "lang_et"},
-  {"name": "Filipino", "value": "lang_tl"},
+  {"name": "Filipino (Pilipino)", "value": "lang_tl"},
-  {"name": "Finnish", "value": "lang_fi"},
+  {"name": "Finnish (Suomalainen)", "value": "lang_fi"},
-  {"name": "French", "value": "lang_fr"},
+  {"name": "French (Français)", "value": "lang_fr"},
-  {"name": "German", "value": "lang_de"},
+  {"name": "German (Deutsche)", "value": "lang_de"},
-  {"name": "Greek", "value": "lang_el"},
+  {"name": "Greek (Ελληνικά)", "value": "lang_el"},
-  {"name": "Hebrew", "value": "lang_iw"},
+  {"name": "Hebrew (עִברִית)", "value": "lang_iw"},
-  {"name": "Hindi", "value": "lang_hi"},
+  {"name": "Hindi (हिंदी)", "value": "lang_hi"},
-  {"name": "Hungarian", "value": "lang_hu"},
+  {"name": "Hungarian (Magyar)", "value": "lang_hu"},
-  {"name": "Icelandic", "value": "lang_is"},
+  {"name": "Icelandic (Íslenska)", "value": "lang_is"},
-  {"name": "Indonesian", "value": "lang_id"},
+  {"name": "Indonesian (Indonesian)", "value": "lang_id"},
-  {"name": "Italian", "value": "lang_it"},
+  {"name": "Italian (Italiano)", "value": "lang_it"},
-  {"name": "Japanese", "value": "lang_ja"},
+  {"name": "Japanese (日本語)", "value": "lang_ja"},
-  {"name": "Korean", "value": "lang_ko"},
+  {"name": "Korean (한국어)", "value": "lang_ko"},
-  {"name": "Latvian", "value": "lang_lv"},
+  {"name": "Latvian (Latvietis)", "value": "lang_lv"},
-  {"name": "Lithuanian", "value": "lang_lt"},
+  {"name": "Lithuanian (Lietuvis)", "value": "lang_lt"},
-  {"name": "Norwegian", "value": "lang_no"},
+  {"name": "Norwegian (Norwegian)", "value": "lang_no"},
-  {"name": "Persian", "value": "lang_fa"},
+  {"name": "Persian (فارسی)", "value": "lang_fa"},
-  {"name": "Polish", "value": "lang_pl"},
+  {"name": "Polish (Polskie)", "value": "lang_pl"},
-  {"name": "Portuguese", "value": "lang_pt"},
+  {"name": "Portugese (Português)", "value": "lang_pt"},
-  {"name": "Romanian", "value": "lang_ro"},
+  {"name": "Romanian (Română)", "value": "lang_ro"},
-  {"name": "Russian", "value": "lang_ru"},
+  {"name": "Russian (русский)", "value": "lang_ru"},
-  {"name": "Serbian", "value": "lang_sr"},
+  {"name": "Serbian (Српски)", "value": "lang_sr"},
-  {"name": "Slovak", "value": "lang_sk"},
+  {"name": "Slovak (Slovák)", "value": "lang_sk"},
-  {"name": "Slovenian", "value": "lang_sl"},
+  {"name": "Slovenian (Slovenščina)", "value": "lang_sl"},
-  {"name": "Spanish", "value": "lang_es"},
+  {"name": "Spanish (Español)", "value": "lang_es"},
-  {"name": "Swahili", "value": "lang_sw"},
+  {"name": "Swahili (Kiswahili)", "value": "lang_sw"},
-  {"name": "Swedish", "value": "lang_sv"},
+  {"name": "Swedish (Svenska)", "value": "lang_sv"},
-  {"name": "Thai", "value": "lang_th"},
+  {"name": "Thai (ไทย)", "value": "lang_th"},
-  {"name": "Turkish", "value": "lang_tr"},
+  {"name": "Turkish (Türk)", "value": "lang_tr"},
-  {"name": "Ukrainian", "value": "lang_uk"},
+  {"name": "Ukranian (Український)", "value": "lang_uk"},
-  {"name": "Vietnamese", "value": "lang_vi"}
+  {"name": "Vietnamese (Tiếng Việt)", "value": "lang_vi"}
 ]

app/static/settings/translations.json

@@ -0,0 +1,58 @@
+{
+    "lang_en": {
+        "search": "Search",
+        "config": "Configuration",
+        "config-country": "Filter Results by Country",
+        "config-country-help": "Note: If enabled, a website will only appear in the search results if it is *hosted* in the selected country.",
+        "config-lang": "Interface Language",
+        "config-lang-search": "Search Language",
+        "config-near": "Near",
+        "config-near-help": "City Name",
+        "config-block": "Block",
+        "config-block-help": "Comma-separated site list",
+        "config-nojs": "Show NoJS Links",
+        "config-dark": "Dark Mode",
+        "config-safe": "Safe Search",
+        "config-alts": "Replace Social Media Links",
+        "config-alts-help": "Replaces Twitter/YouTube/Instagram/etc links with privacy respecting alternatives.",
+        "config-new-tab": "Open Links in New Tab",
+        "config-images": "Full Size Image Search",
+        "config-images-help": "(Experimental) Adds the 'View Image' option to desktop image searches. This will cause image result thumbnails to be lower resolution.",
+        "config-tor": "Use Tor",
+        "config-get-only": "GET Requests Only",
+        "config-url": "Root URL",
+        "config-css": "Custom CSS",
+        "load": "Load",
+        "apply": "Apply",
+        "save-as": "Save As...",
+        "github-link": "View on GitHub"
+    },
+    "lang_es": {
+        "search": "Buscar",
+        "config": "Configuración",
+        "config-country": "Filtrar Resultados por País",
+        "config-country-help": "Nota: Si está habilitado, un sitio web solo aparecerá en los resultados de búsqueda si está alojado en ese país.",
+        "config-lang": "Idioma de Interfaz",
+        "config-lang-search": "Idioma de Búsqueda",
+        "config-near": "Cerca",
+        "config-near-help": "Nombre de la Ciudad",
+        "config-block": "Bloquear",
+        "config-block-help": "Lista de sitios separados por comas",
+        "config-nojs": "Mostrar Enlaces NoJS",
+        "config-dark": "Modo Oscuro",
+        "config-safe": "Búsqueda Segura",
+        "config-alts": "Reemplazar Enlaces de Redes Sociales",
+        "config-alts-help": "Reemplaza los enlaces de Twitter/YouTube/Instagram/etc con alternativas que respetan la privacidad.",
+        "config-new-tab": "Abrir enlaces en una pestaña nueva",
+        "config-images": "Búsqueda de imágenes a tamaño completo",
+        "config-images-help": "(Experimental) Agrega la opción 'Ver imagen' a las búsquedas de imágenes de escritorio. Esto hará que las miniaturas de los resultados de la imagen aparezcan con una resolución más baja.",
+        "config-tor": "Usa Tor",
+        "config-get-only": "GET solo solicitudes",
+        "config-url": "URL raíz",
+        "config-css": "CSS personalizado",
+        "load": "Cargar",
+        "apply": "Aplicar",
+        "save-as": "Guardar como...",
+        "github-link": "Ver en GitHub"
+    }
+}

app/templates/display.html

@@ -20,7 +20,7 @@
     <footer>
         <p style="color: {{ 'var(--whoogle-dark-text)' if config.dark else 'var(--whoogle-text)' }};">
             Whoogle Search v{{ version_number }} ||
-            <a id="gh-link" href="https://github.com/benbusby/whoogle-search">View on GitHub</a>
+            <a id="gh-link" href="https://github.com/benbusby/whoogle-search">{{ translation['github-link'] }}</a>
         </p>
     </footer>
 	<script src="static/js/autocomplete.js"></script>

app/templates/index.html

@@ -53,17 +53,17 @@
 							autocorrect="off"
 							autocomplete="off">
 					</div>
-					<input type="submit" id="search-submit" value="Search">
+					<input type="submit" id="search-submit" value="{{ translation['search'] }}">
 				</div>
 			</form>
 		{% if not config_disabled %}
 			<br/>
-			<button id="config-collapsible" class="collapsible">Configuration</button>
+			<button id="config-collapsible" class="collapsible">{{ translation['config'] }}</button>
 			<div class="content">
 			<div class="config-fields">
 				<form id="config-form" action="config" method="post">
 				<div class="config-div config-div-ctry">
-					<label for="config-ctry">Filter Results by Country: </label>
+					<label for="config-ctry">{{ translation['config-country'] }}: </label>
 					<select name="ctry" id="config-ctry">
 					{% for ctry in countries %}
 						<option value="{{ ctry.value }}"
@@ -74,10 +74,10 @@
 						</option>
 					{% endfor %}
 					</select>
-					<div><span class="info-text"> — Note: If enabled, a website will only appear in the results if it is *hosted* in the selected country.</span></div>
+					<div><span class="info-text"> — {{ translation['config-country-help'] }}</span></div>
 				</div>
 				<div class="config-div config-div-lang">
-					<label for="config-lang-interface">Interface Language: </label>
+					<label for="config-lang-interface">{{ translation['config-lang'] }}: </label>
 					<select name="lang_interface" id="config-lang-interface">
 					{% for lang in languages %}
 					<option value="{{ lang.value }}"
@@ -90,7 +90,7 @@
 					</select>
 				</div>
 				<div class="config-div config-div-search-lang">
-					<label for="config-lang-search">Search Language: </label>
+					<label for="config-lang-search">{{ translation['config-lang-search'] }}: </label>
 					<select name="lang_search" id="config-lang-search">
 					{% for lang in languages %}
 					<option value="{{ lang.value }}"
@@ -103,55 +103,53 @@
 					</select>
 				</div>
 				<div class="config-div config-div-near">
-					<label for="config-near">Near: </label>
+					<label for="config-near">{{ translation['config-near'] }}: </label>
 					<input type="text" name="near" id="config-near" placeholder="City Name" value="{{ config.near }}">
 				</div>
 				<div class="config-div config-div-block">
-					<label for="config-block">Block: </label>
+					<label for="config-block">{{ translation['config-block'] }}: </label>
 					<input type="text" name="block" id="config-block" placeholder="Comma-separated site list" value="{{ config.block }}">
 				</div>
 				<div class="config-div config-div-nojs">
-					<label for="config-nojs">Show NoJS Links: </label>
+					<label for="config-nojs">{{ translation['config-nojs'] }}: </label>
 					<input type="checkbox" name="nojs" id="config-nojs" {{ 'checked' if config.nojs else '' }}>
 				</div>
 				<div class="config-div config-div-dark">
-					<label for="config-dark">Dark Mode: </label>
+					<label for="config-dark">{{ translation['config-dark'] }}: </label>
 					<input type="checkbox" name="dark" id="config-dark" {{ 'checked' if config.dark else '' }}>
 				</div>
 				<div class="config-div config-div-safe">
-					<label for="config-safe">Safe Search: </label>
+					<label for="config-safe">{{ translation['config-safe'] }}: </label>
 					<input type="checkbox" name="safe" id="config-safe" {{ 'checked' if config.safe else '' }}>
 				</div>
 				<div class="config-div config-div-alts">
-					<label class="tooltip" for="config-alts">Replace Social Media Links: </label>
+					<label class="tooltip" for="config-alts">{{ translation['config-alts'] }}: </label>
 					<input type="checkbox" name="alts" id="config-alts" {{ 'checked' if config.alts else '' }}>
-					<div><span class="info-text"> — Replaces Twitter/YouTube/Instagram/Reddit links
+					<div><span class="info-text"> — {{ translation['config-alts-help'] }}</span></div>
-					with Nitter/Invidious/Bibliogram/Libreddit links.</span></div>
 				</div>
 				<div class="config-div config-div-new-tab">
-					<label for="config-new-tab">Open Links in New Tab: </label>
+					<label for="config-new-tab">{{ translation['config-new-tab'] }}: </label>
 					<input type="checkbox" name="new_tab" id="config-new-tab" {{ 'checked' if config.new_tab else '' }}>
 				</div>
 				<div class="config-div config-div-view-image">
-					<label for="config-view-image">Full Size Image Search: </label>
+					<label for="config-view-image">{{ translation['config-images'] }}: </label>
 					<input type="checkbox" name="view_image" id="config-view-image" {{ 'checked' if config.view_image else '' }}>
-					<div><span class="info-text"> — (Experimental) Adds the "View Image" option on desktop to view full size images in search results.
+					<div><span class="info-text"> — {{ translation['config-images-help'] }}</span></div>
-					This will cause image result thumbnails to be lower resolution.</span></div>
 				</div>
 				<div class="config-div config-div-tor">
-					<label for="config-tor">Use Tor: {{ '' if tor_available else 'Unavailable' }}</label>
+					<label for="config-tor">{{ translation['config-tor'] }}: {{ '' if tor_available else 'Unavailable' }}</label>
 					<input type="checkbox" name="tor" id="config-tor" {{ '' if tor_available else 'hidden' }} {{ 'checked' if config.tor else '' }}>
 				</div>
 				<div class="config-div config-div-get-only">
-					<label for="config-get-only">GET Requests Only: </label>
+					<label for="config-get-only">{{ translation['config-get-only'] }}: </label>
 					<input type="checkbox" name="get_only" id="config-get-only" {{ 'checked' if config.get_only else '' }}>
 				</div>
 				<div class="config-div config-div-root-url">
-					<label for="config-url">Root URL: </label>
+					<label for="config-url">{{ translation['config-url'] }}: </label>
 					<input type="text" name="url" id="config-url" value="{{ config.url }}">
 				</div>
 				<div class="config-div config-div-custom-css">
-					<label for="config-style">Custom CSS:</label>
+					<label for="config-style">{{ translation['config-css'] }}:</label>
 					<textarea
 					name="style"
 					id="config-style"
@@ -164,9 +162,9 @@
 					</textarea>
 				</div>
 				<div class="config-div">
-					<input type="submit" id="config-load" value="Load">&nbsp;
+					<input type="submit" id="config-load" value="{{ translation['load'] }}">&nbsp;
-					<input type="submit" id="config-submit" value="Apply">&nbsp;
+					<input type="submit" id="config-submit" value="{{ translation['apply'] }}">&nbsp;
-					<input type="submit" id="config-save" value="Save As...">
+					<input type="submit" id="config-save" value="{{ translation['save-as'] }}">
 				</div>
 				</form>
 			</div>
@@ -176,7 +174,7 @@
 		<footer>
 			<p style="color: {{ 'var(--whoogle-dark-text)' if config.dark else 'var(--whoogle-text)' }};">
 				Whoogle Search v{{ version_number }} ||
-				<a id="gh-link" href="https://github.com/benbusby/whoogle-search">View on GitHub</a>
+				<a id="gh-link" href="https://github.com/benbusby/whoogle-search">{{ translation['github-link'] }}</a>
 			</p>
 		</footer>
 	</body>

test/test_misc.py

@@ -1,5 +1,6 @@
 from cryptography.fernet import Fernet
 
+from app import app
 from app.utils.session import generate_user_key, valid_user_session
 
 
@@ -15,6 +16,18 @@ def test_valid_session(client):
         assert valid_user_session(session)
 
 
+def test_valid_translation_keys(client):
+    valid_lang_keys = [_['value'] for _ in app.config['LANGUAGES']]
+    en_keys = app.config['TRANSLATIONS']['lang_en'].keys()
+    for translation_key in app.config['TRANSLATIONS']:
+        # Ensure the translation is using a valid language value
+        assert translation_key in valid_lang_keys
+
+        # Ensure all translations match the same size/content of the original
+        # English translation
+        assert app.config['TRANSLATIONS'][translation_key].keys() == en_keys
+
+
 def test_query_decryption(client):
     # FIXME: Handle decryption errors in search.py and rewrite test
     # This previously was used to test swapping decryption keys between