Upgrading to V5.4 from e2guardian v5.3

If you are upgrading from v4 please read notes/Upgrading_to_V5 first.

There are changes to both the configuration files, mainly in the definition
of auth plugins, 

Both the e2guardian.conf and e2guardianf1.conf contain some new list definitions.  

Some new list types are introduced, including maplist, ipmaplist and mapport all
of which are used to map users/ip/ports to filter groups
Details of the new list types definition are in notes/V5_list_definition.pdf. 

A number of settings have been moved into Storyboard config files. 
These are noted in the example e2guardian.conf and e2guardianf1.conf files.

For details on Storyboard syntax see notes/V5_Storyboard.pdf


UPGRADE PROCEDURE

Before upgrading make a backup of all your existing configuration & lists. You will
may need to refer to these after the install.

Install E2guardian V5.4 in a clean area.

Main configuration files.

You should be able to use an edited versions of your v5.3 e2guardian.conf and e2guardianf1.conf files.

Changes needed to e2guardian.conf
---------------------------------

New logfile formats - see notes/New_log_fields_in_log_format7-8.pdf

# 7 = Same as 5, but with searchterms and EXTFLAGS added
#     See notes/New_log_fileds_in_log_format7-8.pdf for details
# 8 = Same as 7, but with server field blanked
logfileformat = 8

New useoriginalip option.

# This option only applies when request is transparent (http or https),
# when no upstream proxy is used, and where it is possible to detect
# the original destination ip & port
# When enabled the upstream request will be directed at the original ip and port# and no DNS lookup will be performed.
# This solves the 'snapchat' issue and also should increase speed of connection.
# Currently this only works on linux systems.  BSD developers, PLEASE HELP fix
# this for BSD, pfsense etc!
# default = off
useoriginalip = on

originalip option is no longer supported - will be ignored in v5.4

Filtergrouplist is replaced with maplist

Replace this:-
filtergroupslist = '@E2CONFDIR@/lists/filtergroupslist'
with:-
maplist = 'name=defaultusermap, path=@E2CONFDIR@/lists/filtergroupslist'

Add defines for nomitm lists - these are lists of sites that should never be mitm (where mitm cannot work) e.g. Sites only used by apps that do not follow normal certificate rules etc. Sites in these lists will be treated as non-mitm.

sitelist = 'name=nomitm,path=@E2CONFDIR@/lists/common/nomitmsitelist'
ipsitelist = 'name=nomitm,path=@E2CONFDIR@/lists/common/nomitmsiteiplist'

Add defines for browser list - it is used to determine whether to return a block page with https.
Non-browser user-agents cannot understand block page and often refuse to be mitm.

regexpboollist = 'name=browser,path=@E2CONFDIR@/lists/common/browserregexplist'

Add maplists for any auth plugins you are using - Note you must also replace the authplugins/*.conf
files with the new v5.4 examples.

#ipmaplist = 'name=ipmap, path=@E2CONFDIR@/lists/authplugins/ipgroups'

#maplist = 'name=portmap, path=@E2CONFDIR@/lists/authplugins/portgroups'

New logclientnameandip option:-
# Enable logging of client hostname and its IP
# If off the hostname will be logged instead of IP
# Applies only to log formats 5, 6, 7 & 8.
# on|off (defaults to on)
# Needs to be turned off if you are using sarg log analysis.
logclientnameandip = on

# use dash ('-') instead of blank fields in log
# This is essential for space delimited log formats, and makes all log types easier to read
# But can be turned off if this causes a problem with log analysis
# on|off (defaults to on)
usedashforblank = on


Changes needed to e2guardianf1.conf
-----------------------------------

Add bannedtimelist - times when no access is allowed
# To activate a storyboard change is required - see examplef1.story
timelist = 'name=bannedtimes,messageno=122,path=@E2CONFDIR@/lists/bannedtimelist'

Add blankettimelist - times when blanket block is applied
# To activate a storyboard change is required - see examplef1.story
timelist = 'name=blankettimes,messageno=122,path=@E2CONFDIR@/lists/blankettimelist'

Add response header list defines if required

# Response HTTP header rules:
# Optional lists for modification or removal of HTTP response headers.
# Format for reponseheaderregexplist is one rule per line, similar to
# content/URL modifications.
# Headers are matched/replaced on a line-by-line basis, not as a contiguous
# block.
# Use for example, to remove protocol upgrade requests.
regexpreplacelist = 'name=reponseheadermods,path=@E2CONFDIR@/lists/responseheaderregexplist'

Add searchexceptionregexplist define

# search terms exception (overide) list
# Used to prevent urls such os completetion suggestion requests from being detected
# as search requests
regexpboollist = 'name=searchtermexceptions, path=@E2CONFDIR@/lists/common/searchexceptionregexplist'

Storyboards
-----------

Replace the old common.story with the new v5.4 version.

Replace the old preauth.story with the new one.  Note that group assignment is 
now storyboarded in this file allowing more flexible group assignment.

Using the new v5.4 examplef1.story as a template create a new fn.story for each filtergroup, 
actioning any blanket block required and refering to your old fn.story file(s).

Remove the line 'function(checknobypasslists)' from your site.story file.

Lists
-----

You may wish to reorganize your existing lists along along the same lines as 
suggested in the list/README, but this is optional.

Copy the following list files (new or revised in v5.4) into your working 
directory(ies) and then edit them to your requirements:-

common/nomitmsitelist
common/nomitmsiteiplist
common/browserregexplist
example.group/bannedphraselist
example.group/weightedphraselist
example.group/exceptionphraselist
example.group/oldbannedphraselist
example.group/oldweightedphraselist
example.group/oldexceptionphraselist
example.group/bannedtimelist
example.group/blankettimelist
example.group/responseheaderregexplist
common/nologurllist
common/nologsitelist
common/nologregexpurllist
common/nologextensionlist
common/searchexceptionregexplist

Note that phrase list paths have changed so that they are organised by language.

First run
---------

On the initial test run look at the standard output of e2guardian.  This will show any missing lists as well as those
which are never used by the storyboarding.

Revised 10 August 2020

