Merge documentation improvements from branch dev_v0.3 into master

README.md

@@ -1,9 +1,25 @@
-# pihole-bl-msft-telemetry-bsi
-Pi-Hole blocklist for hosts involved in Microsoft Windows telemetry, as documented by BSI Bund in Germany.
+# A vast and useful collection of Pi-Hole block lists
+
+Here I publish a collection of Pi-Hole blocklists, including a blocklist disallowing well known
+Microsoft(tm) Windows(R) Telemetry hosts, as documented by BSI Bund in Germany in their project "SiSyPHuS Win10".
+The BSI is the "German Federal Bureau of Security in Information Technology" (in German: Bundesamt für Sicherheit in der Informationstechnik).
 
 Thanks to:  
-- [BSI Bund](https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Sicherheit/SiSyPHus/Telemetrie-Endpunkte_Windows10_Build_Build_21H2.html)  
-- [Karsten Neß](https://www.privacy-handbuch.de/autoren.htm)  
+- [BSI Bund](https://www.bsi.bund.de/)  
+- [Karsten Neß, principal author of "The Privacy Handbook"](https://www.privacy-handbuch.de/autoren.htm)  
+
+
+You might want to refer to these documents published by the BSI:
+
+[Analysis of Telemetry component in Windows 10, V1.2](https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Sicherheit/SiSyPHus/Analyse_Telemetriekomponente_1_2.html)
+
+[Telemetry end-points in Windows 10 Build 1809](https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Sicherheit/SiSyPHus/Telemetrie-Endpunkte_Windows10_Build_1809.html)
+
+[Telemetry end-points in Windows 10 Build 21H2](https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Sicherheit/SiSyPHus/Telemetrie-Endpunkte_Windows10_Build_Build_21H2.html)
+
+[Deactivation of the Telemetry component in Windows 10 Build 21H2 V1.0](https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Sicherheit/SiSyPHus/E20172000_BSI_Win10_AFUNKT_TELE_DEAKTIVIEREN_v1_0.html)
+
+[Windows 10: BSI offers tool for Telemetry monitoring, July 7, 2022](https://www.bsi.bund.de/DE/Service-Navi/Presse/Alle-Meldungen-News/Meldungen/Tool_Telemetrie-Monitoring_220719.html)
 
 
 I also added my list of 68 blocklists that I found on the Interwebs by craft of my Google-Fu.  
@@ -13,49 +29,73 @@ Thanks to:
 - [hagezi](https://github.com/hagezi)  
 - [sjhgvr](https://oisd.nl/)  
 
+Furthermore, I included commonly whitelisted domains from the Pi-Hole Discourse page (see at the bottom)
+as well as some of my own personal whitelist entries.  This whitelist is far from complete, so some sites
+you use might not work.  Please let me know, so that I can include them.  Please post this information
+also to the Pi-Hole discourse list.
+
+It should be noted that currently, the import process will completely replace all blacklist and whitelist entries
+in your Pi-Hole installation from the files I supply here.  This will probably be improved in a future version.
+For progress on this necessary enhancement, see also [Issue #7](https://github.com/pschneider1968/pihole-bl-msft-telemetry-bsi/issues/7)
+
 
-Usage:  
+**Usage:**
 
     cd /etc/pihole
     git clone https://github.com/pschneider1968/pihole-bl-msft-telemetry-bsi.git
     cd pihole-bl-msft-telemetry-bsi
     sh refresh_all.sh
 
-Install a crontab like this with crontab -e
+
+Install a crontab like this with `crontab -e` to regularly update from my repo at 0:40 AM in the night:
 
     40 0 * * * sh /etc/pihole/pihole-bl-msft-telemetry-bsi/refresh_all.sh merge
 
 
-The blacklists and whitelists are loaded with the supplied scripts. Also, if you have entered the blocklists manually, one entry of them
-is a pointer to the file `msft_telemetry_bsi.txt` here in this GitHub repo, so this will be included in the refresh processing when it has changed here.  
+The big domain blocklist from file `list_of_blocklists.txt` as well as the blacklists and whitelists are
+loaded with the supplied scripts.  One of those list entries is a pointer to the file `msft_telemetry_bsi.txt`
+here in this GitHub repository, which contains the list of Microsoft hosts involved in Windows telemetry,
+as documented by the BSI.
+
+This list will thus be included in the refresh processing when it has changed here in my repo.
+
 
-The import process can be run in four different modes: ADD, MERGE, DELETE and FULL. Please call the script import_lists.sh with the parameter HELP to learn more:
+The import process can be run in four different modes: ADD, MERGE, DELETE and FULL.
+Please call the script import_lists.sh with the parameter HELP to learn more:
 
 
+
     $ sh import_lists.sh HELP
     import_lists.sh v0.2 (c) 2022 Peter Schneider, provided under MIT License
 
     Synopsis: import_lists.sh [MODE]
 
-    This script will import the contents of the supplied file "list_of_blocklists.txt" into your Pi-Hole Gravity DB,
-    where MODE may be one of:
+    This script will import the contents of the supplied file "list_of_blocklists.txt" into your
+    Pi-Hole Gravity DB, where MODE may be one of:
+
+    - HELP:    Display this help info
+
+    - ADD:     Only add new lists, don't do anything to existing lists.  This is the recommended mode
+               of operation when you have other sources for your block lists, too, other than my repo.
+               It is also the default when no MODE is specified.
+
+    - MERGE:   Add new lists, disable missing ones, re-enable disabled existing lists if they are in the
+               import file.  This retains group assignments on existing list entries.  This is the recommended
+               mode of operation when my repo is the ONLY source of block lists for your Pi-Hole installation.
+
+    - DELETE:  Add new lists, delete missing ones, re-enable disabled existing lists if they are in the
+               import file.  Group assignments on deleted groups are of course lost, and they cannot
+               just be re-enabled again, but will be newly imported when they happen to be in the
+               next version of the import file again.
+
+    - FULL:    Fully replace all existing list entries in Gravity DB with the imported ones.
+               Group assignments are thus lost.  That means that before inserting anything from the
+               import file, everything is deleted in the Gravity DB.
 
-    - HELP:    display this help info
-    - ADD:     only add new lists, don't do anything to existing lists. This is the recommended mode of operation
-                when you have other sources for your block lists, too, other than my repo.
-                It is also the default when no MODE is specified.
-    - MERGE:   add new lists, disable missing ones, re-enable disabled existing lists if they are in the import file.
-                This retains group assignments on existing list entries. This is the recommended mode of operation
-                when my repo is the ONLY source of block lists for your Pi-Hole installation.
-    - DELETE:  add new lists, delete missing ones, re-enable disabled existing lists if they are in the import file.
-                Group assignments on deleted groups are of course lost, and they cannot just be re-enabled again,
-                but will be newly imported when they happen to be in the next import file again.
-    - FULL:    fully replace all existing list entries in Gravity DB with the imported ones. Group assignments are thus lost.
-                That means that before inserting anything from the import file, everything is deleted in the Gravity DB.
 
 
 I will try to check for updates and new lists on a regular basis, but I can't promise anything.
-As of today (Dec 19, 2022) the total number of blocked unique domains from all these lists is 5838819.
+As of today (Dec 22, 2022) the total number of blocked unique domains from all these lists is 6775012.
 
 If something does not work, or you suspect false positives, check against the commonly whitelisted domains at [Pi-Hole Discourse](https://discourse.pi-hole.net/t/commonly-whitelisted-domains/212)  
 

import_lists.sh

@@ -67,30 +67,41 @@ if [ -z "$MODE" ]; then         # default mode
     MODE='ADD'
 fi
 
-if [ "$MODE" = "HELP" -o "$MODE" = "--HELP" -o "$MODE" = "-H" -o "$MODE" = "-?" -o "$MODE" = "/?" ]; then
-    echo 
+if [ "$MODE" = "HELP" -o "$MODE" = "--HELP" -o "$MODE" = "-H" -o "$MODE" = "-?" -o "$MODE" = "/?" ]
+then
+
+    echo
     echo "Synopsis: import_lists.sh [MODE]"
     echo
-    echo "This script will import the contents of the supplied file \"list_of_blocklists.txt\" into your Pi-Hole Gravity DB,"
-    echo "where MODE may be one of:"
+    echo "This script will import the contents of the supplied file \"list_of_blocklists.txt\" into your"
+    echo "Pi-Hole Gravity DB, where MODE may be one of:"
     echo
     echo "   - HELP:    display this help info"
-    echo "   - ADD:     only add new lists, don't do anything to existing lists. This is the recommended mode of operation"
-    echo "              when you have other sources for your block lists, too, other than my repo."
+    echo
+    echo "   - ADD:     Only add new lists, don't do anything to existing lists.  This is the recommended mode"
+    echo "              of operation when you have other sources for your block lists, too, other than my repo."
     echo "              It is also the default when no MODE is specified."
-    echo "   - MERGE:   add new lists, disable missing ones, re-enable disabled existing lists if they are in the import file."
-    echo "              This retains group assignments on existing list entries. This is the recommended mode of operation"
-    echo "              when my repo is the ONLY source of block lists for your Pi-Hole installation."
-    echo "   - DELETE:  add new lists, delete missing ones, re-enable disabled existing lists if they are in the import file."
-    echo "              Group assignments on deleted groups are of course lost, and they cannot just be re-enabled again,"
-    echo "              but will be newly imported when they happen to be in the next import file again."
-    echo "   - FULL:    fully replace all existing list entries in Gravity DB with the imported ones. Group assignments are thus lost."
-    echo "              That means that before inserting anything from the import file, everything is deleted in the Gravity DB."
-    echo 
+    echo
+    echo "   - MERGE:   Add new lists, disable missing ones, re-enable disabled existing lists if they are in"
+    echo "              the import file.  This retains group assignments on existing list entries. This is the recommended"
+    echo "              mode of operation when my repo is the ONLY source of block lists for your Pi-Hole installation."
+    echo
+    echo "   - DELETE:  Add new lists, delete missing ones, re-enable disabled existing lists if they are in the"
+    echo "              import file.  Group assignments on deleted groups are of course lost, and they cannot"
+    echo "              just be re-enabled again, but will be newly imported when they happen to be in the"
+    echo "              next version of the import file again."
+    echo
+    echo "   - FULL:    Fully replace all existing list entries in Gravity DB with the imported ones."
+    echo "              Group assignments are thus lost.  That means that before inserting anything from the"
+    echo "              import file, everything is deleted in the Gravity DB."
+    echo
+
 elif [ ! "$MODE" = "ADD" -a ! "$MODE" = "MERGE" -a ! "$MODE" = "DELETE" -a ! "$MODE" = "FULL" ]
 then
+
     echo "ERROR: Unknown mode $MODE, please use the parameter \"HELP\" for information on script usage!"
     exit $ERROR
+
 else
     echo "Mode of import operation is $MODE"
 fi

whitelist_exact.txt

@@ -5,6 +5,7 @@ presence.teams.microsoft.com
 disqus.com
 www.disqus.com
 user-rating.burda-forward.de
+boards.greenhouse.io
 
 
 # URL shortener