TLS Data Gatherer

This project aims for extracting relevant data of different technologies from Qualys reports. It is currently intended for servers using Apache HTTP Server.

Stages

1. Qualys output files (.csv) are formatted according to our convenience. The Evidence column is divided into three (Extended Evidence, Current Value, Expected Value; we will make use of Current Value the most).
2. A row from the original .csv report represents the evaluation of a specific control for a specific server. This would be better evaluated if for the same server, all data from all controls were available within a single row. This stage merges data from all controls for the same server in a single row.
3. Detection of all IP:port configurations, including those labelled as VirtualHosts. Separation of columns for each Apache instance by IP:port pairs, each following its own settings.
4. Available TLS versions for connections with each server are calculated from the Current Values. Since enabling/disabling versions is different per technology, each needs to have their own rulesets defined.
5. From the list of TLS versions calculated on Stage 4 and the list of Cipher Suites configured for a server, the Compliance Status and Safety for both the TLS versions and the Cipher Suites is evaluated, following criteria based on the Santander Cryptography Standard.
6. (Potentially optional in the future) The final output .csv file is obtained from removing all redundant columns from the initial .csv file, unneeded after the extraction of the relevant data. The plotting of the TLS version & Cipher Suites compliance and safety status (be it: global, or by environments) is given as an output as well.

NOTE: The code from this repo is still a work in progress. It is intended for Qualys .csv reports from the Policies CS-ST-034-GlobalCryptography_Apache. Multiple technologies are yet to be added.

How to create a Policy Report

The policy report is the input to our process. In order to get the .csv files, two elements need to be created within Qualys:

1. A Policy for the technology (for now, Apache HTTP Server) with the corresponding controls. This code assumes the following CIDs are included:

   • 16086 - Status of the 'SSLEngine' derivative within the Apache configuration files (server config and virtual host)
   • 9798 - Status of "Header" setting within virtualhost whose "SSLEngine" is set "on"
   • 9799 - List of VirtualHost elements whose "SSLEngine" is set "on" and no "Header" set neither at server level nor at Virtual Host level
   • 10838 - Status of 'SSLProtocol' at server level
   • 10840 - Status of "SSLCipherSuite" settings
   • 7640 - Status of the 'Listen' directive in the Apache configuration file on the host
   • 19505 - Status of the 'VirtualHost' directive in the apache configuration file

2. A Policy Report. Data from the assets included in the Policy are tested against its controls. Regardless of the Pass condition imposed, evidences for each Pass/Fail instance are retrieved.

The first step is to switch to the "Policy Compliance" module, if not already there.

Policy creation

1. From "Policy Compliance", select "Policies". From the tabs available, a second "Policies" will appear as selected.

2. Select: New > Policy > Create from Scratch

3. Search technologies: for the current Apache functionality, search for Apache HTTP Server.

4. Choose Target Hosts from either Asset Groups or Tags (the repor is generated by evaluating through a Tag, given to all assets with Apache HTTP Server instances).

5. Name your policy. We will pick for the example "Apache_HTTP_Server_TDG_Policy".

6. An empty Section for the Policy will be present upon creating it. Each Section may store one or multiple Controls. For adding new sections: Add a new Section.

7. For adding Controls on each Section: Add Controls > Search (by CIDs: 10838**) > Add. Repeat for all Controls required on each Section.

   ** This particular CID corresponds to "Status of 'SSLProtocol' at server level".

8. Save the Policy

Report generation

1. From "Policy Compliance", select "Reports". From the tabs available, a second "Reports" will appear as selected.
2. Select: New > Compliance Report > Policy Report
3. Name your report. We will pick for the example "Apache_HTTP_Server_TDG_Report".
4. For a Report Template, pick "Policy Report Template". It is a generic template that should be available on all Qualys distributions.
5. For a Report Format, select "Comma-Separated Value (CSV)".
6. For a Policy, select the one created earlier (it is Apache_HTTP_Server_TDG_Policy in our example). The controls within this policy will be evaluated.
7. Finally, press "Run". It will take some time. Once it finishes, locate the finished Report in the list. Select: Quick Actions > Download.

How to Use

For a .csv document (the Policy Report taken from Qualys) and technology choice (for now, the option would be "apache") execute the following command line:

python tls_data_gatherer.py <file_name.csv> <tech>

Example:

python tls_data_gatherer.py "./data/Compliance_Report_Apache_0123456.csv" "apache"

Outputs

Six .csv documents will be generated, each corresponding to the end of a phase, plus final (stage 6) .csv reports for each entity/unit detected, plus several general plots, plus plots for each entity. Outputs are:

1. <file_name>_Stage1_csvCleanse.csv - Corresponds to the fixed .csv, removing all undesired information and readjusting the format.
2. <file_name>_Stage2_RowsMerged.csv - All rows corresponding to a Host IP & Instance, each row containing information for a Qualys control, are grouped together.
3. <file_name>_Stage3_VirtualHostSeparation.csv - Each instance row is separated into IP:port-specific rows. Name of the file is motivated by VirtualHosts being the main use case this separation process is executed for.
4. <file_name>_Stage4_TLSVersionsInterpreted.csv - Enabled TLS versions are interpreted from the Qualys control results.
5. <file_name>_Stage5_CriteriaEstablished.csv - TLS version data is categorized in terms of Compliance and Safety.
6. <file_name>_Stage6_OutputData.csv - Similar conclusions to Stage 5, but without unneeded data, making it lighter and much easier to read.
7. <file_name>_<unit>_Stage6_OutputData.csv - Stage 6 data specific for each .
8. <file_name>_plot_global_<type>.png - Different global plots.
9. <file_name>_plot_<unit>_<type>.png - Different plot types per unit-specific analysis.

Adding more technologies into the tool

This code is organized around six stages. The addition of new technologies (as IIS, Tomcat, etc) needs to take into account the following steps:

1. Add the new technology to process in the techs_available array in config.py, as well as the parameters needed for that tecnology. So far, there are three parameters:

   • ciphersuites: string, corresponds to the column storing Cipher Suites information for the technology. See Step 5 for more information.
   • dictionary: string, corresponds to the naming the corresponding dictionary has (e.g. dictionary="Apache corresponds to dictionary_Apache.json).

2. The addition of the config dictionary in config_dictionaries directory (e.g. dictionary_Apache.json) should include all the columns of the report intended for the processing. All entries should be interpreted as str.

3. In the folder stage2_row_merging, add a new section for the new tecnology in the function column_controls_adder (file: column_controls_adder.py). This function is intended to group all entries of controls evaluated for the same host, into one single row. Stage 2 in particular processes the following four columns per control, using column_controls_adder:

   • Control - {tag}
   • Status - {tag}
   • Current Value(s) - {tag}
   • Extended Evidence(s) - {tag}

   For Apache HTTP Server, we used CID 10840, and added the tag SSLCipherSuite to the four required columns per control. This is also the value we placed in the ciphersuites field for Step 1.

4. Define an aggregation functions dictionary in folder config_dictionaries equivalent to dictionary_aggr_functions_Apache.json where you define how each column will be aggregated during the process. Original columns from the pre-processing .csv file should be interpreted as first, whereas columns that will be generated by controls should be interpreted as sum.

5. In the folder stage4_tls_versions, in file tls_version_extraction_enabled.py, add a new section for the new technology. This function is intended to extract the TLS information from the designated column for its storage. In the case of Apache HTTP Server, we used the Current Value - SSLProtocol column. The column Technology may also be considered for default behavior of said technology.

6. In the folder stage5_criteria, in file tls_version_compliance_filter.py, take into account that rules may vary depending on technology. Add as many rules as needed.

7. In the folder stage5_criteria, in file tls_version_safety_filter.py, take into account that rules may vary depending on technology. Add as many rules as needed.