Port FakeNet to Python 3 (#157)

* Port FakeNet to Python 3

* Migrate diverters, listeners and other
components to Python 3
* Retire BITS listener for now
* Fix hard-coded FakeNet path for linux
platform in test.py
* Update README and developement documentation
* Add copyright notice to modified files

* Update FakeNet-NG version

Author: tinajn

CHANGELOG.txt

@@ -1,3 +1,11 @@
+Version 3.0 (alpha)
+-----------
+* Migrate diverters, listeners and other components to Python 3
+* Retire BITS listener for now
+* Fix hard-coded FakeNet path for linux platform in test.py
+* Update README and developement documentation
+* Add copyright notice to modified files
+
 Version 1.4.13
 --------------
 * Port test scripts to Python 3

README.md

@@ -1,16 +1,3 @@
-Development Suspended
-=====================
-
-The FLARE Team must suspend development and maintenance of FakeNet-NG for the
-time being.
-
-FLARE has opted to indicate the project status here instead of archiving the
-project. This will allow users and maintainers to continue to log issues
-documenting valuable information about problems, troubleshooting, and
-work-arounds.
-
-Original Documentation Follows
-==============================
      ______      _  ________ _   _ ______ _______     _   _  _____
     |  ____/\   | |/ /  ____| \ | |  ____|__   __|   | \ | |/ ____|
     | |__ /  \  | ' /| |__  |  \| | |__     | |______|  \| | |  __
@@ -20,7 +7,7 @@ Original Documentation Follows
 
            D   O   C   U   M   E   N   T   A   T   I   O   N
 
-FakeNet-NG is a next generation dynamic network analysis tool for malware
+FakeNet-NG 3.0 (alpha) is a next generation dynamic network analysis tool for malware
 analysts and penetration testers. It is open source and designed for the latest
 versions of Windows (and Linux, for certain modes of operation). FakeNet-NG is
 based on the excellent Fakenet tool developed by Andrew Honig and Michael
@@ -57,10 +44,6 @@ analysis machine.
 
 Installing module
 -----------------
-
-Installation on Windows requires the following dependency:
- * [Microsoft Visual C++ Compiler for Python 2.7](https://aka.ms/vcpython27)
-
 Installation on Linux requires the following dependencies:
  * Python pip package manager (e.g. python-pip for Ubuntu).
  * Python development files (e.g. python-dev for Ubuntu).
@@ -94,7 +77,7 @@ Finally if you would like to avoid installing FakeNet-NG and just want to run it
 as-is (e.g. for development), then you would need to obtain the source code and
 install dependencies as follows:
 
-1) Install 64-bit or 32-bit Python 2.7.x for the 64-bit or 32-bit versions
+1) Install 64-bit or 32-bit Python 3.7.x for the 64-bit or 32-bit versions
    of Windows respectively.
 
 2) Install Python dependencies:
@@ -116,7 +99,7 @@ install dependencies as follows:
 Execute FakeNet-NG by running it with a Python interpreter in a privileged
 shell:
 
-    python fakenet.py
+    python -m fakenet.fakenet
 
 Usage
 =====
@@ -133,12 +116,12 @@ parameter to get simple help:
      | | / ____ \| . \| |____| |\  | |____   | |      | |\  | |__| |
      |_|/_/    \_\_|\_\______|_| \_|______|  |_|      |_| \_|\_____|
 
-                             Version  1.0
+                             Version  3.0 (alpha)
       _____________________________________________________________
                        Developed by FLARE Team
-        Copyright (C) 2016-2022 Mandiant, Inc. All rights reserved.
+        Copyright (C) 2016-2023 Mandiant, Inc. All rights reserved.
       _____________________________________________________________
-    Usage: fakenet.py [options]:
+    Usage: python -m fakenet.fakenet [options]:
 
     Options:
       -h, --help            show this help message and exit
@@ -188,7 +171,7 @@ and an HTTP connection:
      | | / ____ \| . \| |____| |\  | |____   | |      | |\  | |__| |
      |_|/_/    \_\_|\_\______|_| \_|______|  |_|      |_| \_|\_____|
 
-                             Version  1.0
+                             Version  3.0 (alpha)
       _____________________________________________________________
                        Developed by FLARE Team
         Copyright (C) 2016-2022 Mandiant, Inc. All rights reserved.

docs/srs.md

@@ -116,7 +116,7 @@ currently implemented in any FakeNet-NG release), and `Auto`.
 ### Configuration Settings and Sections
 The configuration file used on each platform must conform to the format used by
 the others. As of this writing, it is based on the Python
-[ConfigParser](https://docs.python.org/2/library/configparser.html) package,
+[ConfigParser](https://docs.python.org/3/library/configparser.html) package,
 which is similar to an INI file.
 
 ### Quick Glossary of Miscellaneous Terms

docs/testing.md

@@ -127,7 +127,7 @@ into `pytest`, that is fine, but the current script serves.
 The salient peculiarity of `test.py` is that on Windows it requires you to
 install FakeNet and it then runs the global `fakenet.exe` script entry point
 available via the Windows path environment variable; and on Linux, it executes
-`python fakenet.py` directly. Alas, this is for purely undocumented and
+`python -m fakenet.fakenet` directly. Alas, this is for purely undocumented and
 forgettable reasons. It shouldn't be too difficult to make these consistent if
 that becomes important to someone.
 
@@ -210,7 +210,8 @@ modes, and clients:
 
 In each combination, FakeNet must be tested against the full range of
 applicable tests in the Automated Test Suite provided by the test script
-`test/test.py`.
+`test/test.py`. It should be noted that some tests are expected to output
+errors such as socket errors even when the tests are successful.
 
 As of this writing, the Manual Test Suite must also be executed if the FakeNet
 feature set is to be fully exercised for quality assurance of a given release.

fakenet/configs/CustomProviderExample.py

@@ -1,3 +1,5 @@
+# Copyright (C) 2016-2023 Mandiant, Inc. All rights reserved.
+
 import socket
 
 # To read about customizing HTTP responses, see docs/CustomResponse.md
@@ -14,7 +16,7 @@ def HandleRequest(req, method, post_data=None):
         The HTTP post data received by calling `rfile.read()` against the
         BaseHTTPRequestHandler that received the request.
     """
-    response = 'Ahoy\r\n'
+    response = b'Ahoy\r\n'
 
     if method == 'GET':
         req.send_response(200)
@@ -51,8 +53,8 @@ def HandleTcp(sock):
         if not data:
             break
 
-        resp = raw_input('\nEnter a response for the TCP client: ')
-        sock.sendall(resp)
+        resp = input('\nEnter a response for the TCP client: ')
+        sock.sendall(resp.encode())
 
 
 def HandleUdp(sock, data, addr):
@@ -68,5 +70,5 @@ def HandleUdp(sock, data, addr):
         The host and port of the remote peer
     """
     if data:
-        resp = raw_input('\nEnter a response for the UDP client: ')
-        sock.sendto(resp, addr)
+        resp = input('\nEnter a response for the UDP client: ')
+        sock.sendto(resp.encode(), addr)

fakenet/configs/default.ini

@@ -178,7 +178,6 @@ BlackListPortsUDP: 67, 68, 137, 138, 443, 1900, 5355
 # * Webroot          - Set webroot path for HTTPListener.
 # * DumpHTTPPosts    - Store HTTP Post requests for the HTTPListener.
 # * DumpHTTPPostsFilePrefix - File prefix for the stored HTTP Post requests used by the HTTPListener.
-# * BITSFilePrefix   - File prefix for the stored BITS uploads used by the BITSListener.
 # * TFTPFilePrefix   - File prefix for the stored tftp uploads used by the TFTPListener.
 # * DNSResponse      - IP address to respond with for A record DNS queries. (DNSListener)
 # * NXDomains        - A number of DNS requests to ignore to let the malware cycle through
@@ -206,7 +205,7 @@ Enabled:     True
 Protocol:    TCP
 Listener:    ProxyListener
 Port:        38926
-Listeners:   HTTPListener, RawListener, FTPListener, DNSListener, POPListener, SMTPListener, TFTPListener, IRCListener, BITSListener
+Listeners:   HTTPListener, RawListener, FTPListener, DNSListener, POPListener, SMTPListener, TFTPListener, IRCListener
 Hidden:      False
 
 [ProxyUDPListener]

fakenet/diverters/debuglevels.py

@@ -1,3 +1,5 @@
+# Copyright (C) 2016-2023 Mandiant, Inc. All rights reserved.
+
 # Debug print levels for fine-grained debug trace output control
 DNFQUEUE = (1 << 0)     # netfilterqueue
 DGENPKT = (1 << 1)      # Generic packet handling
@@ -39,4 +41,4 @@
     DMISC: 'MISC',
 }
 
-DLABELS_INV = {v.upper(): k for k, v in DLABELS.iteritems()}
+DLABELS_INV = {v.upper(): k for k, v in DLABELS.items()}

fakenet/diverters/diverterbase.py

@@ -1,3 +1,5 @@
+# Copyright (C) 2016-2023 Mandiant, Inc. All rights reserved.
+
 import os
 import abc
 import sys
@@ -10,7 +12,7 @@
 import subprocess
 from . import fnpacket
 from . import fnconfig
-from debuglevels import *
+from .debuglevels import *
 from collections import namedtuple
 from collections import OrderedDict
 
@@ -92,7 +94,7 @@ def first_packet_new_session(self):
                     (self.pkt.dst_ip, self.pkt.dport))
 
 
-class DiverterPerOSDelegate(object):
+class DiverterPerOSDelegate(object, metaclass=abc.ABCMeta):
     """Delegate class for OS-specific methods that FakeNet-NG implementors must
     override.
 
@@ -105,7 +107,6 @@ class DiverterPerOSDelegate(object):
         check_gateways (currently only a warning)
         check_dns_servers (currently only a warning)
     """
-    __metaclass__ = abc.ABCMeta
 
     @abc.abstractmethod
     def check_active_ethernet_adapters(self):
@@ -332,7 +333,7 @@ def isHidden(self, proto, port):
 
     def getPortList(self, proto):
         if proto in self.protos:
-            return self.protos[proto].keys()
+            return list(self.protos[proto].keys())
         return []
 
     def intersectsWithPorts(self, proto, ports):
@@ -540,7 +541,7 @@ def __init__(self, diverter_config, listeners_config, ip_addrs,
         stringlists = ['HostBlackList']
         self.configure(diverter_config, portlists, stringlists)
         self.listeners_config = dict((k.lower(), v)
-                                     for k, v in listeners_config.iteritems())
+                                     for k, v in listeners_config.items())
 
         # Local IP address
         self.external_ip = socket.gethostbyname(socket.gethostname())
@@ -773,7 +774,7 @@ def parse_listeners_config(self, listeners_config):
 
         #######################################################################
         # Populate diverter ports and process filters from the configuration
-        for listener_name, listener_config in listeners_config.iteritems():
+        for listener_name, listener_config in listeners_config.items():
 
             if 'port' in listener_config:
 
@@ -1238,6 +1239,12 @@ def formatPkt(self, pkt, pid, comm):
         Returns:
             A str containing the log line
         """
+        if pid == None:
+            pid = 'None'
+
+        if comm == None:
+            comm = 'None'
+
         logline = ''
 
         if pkt.proto == 'UDP':

fakenet/diverters/fnconfig.py

@@ -1,3 +1,5 @@
+# Copyright (C) 2016-2023 Mandiant, Inc. All rights reserved.
+
 class Config(object):
     """Configuration primitives.
 
@@ -19,7 +21,7 @@ def configure(self, config_dict, portlists=[], stringlists=[]):
             2.) Turn string lists into arrays for quicker access
             3.) Expand port range specifications
         """
-        self._dict = dict((k.lower(), v) for k, v in config_dict.iteritems())
+        self._dict = dict((k.lower(), v) for k, v in config_dict.items())
 
         for entry in portlists:
             portlist = self.getconfigval(entry)
@@ -51,8 +53,8 @@ def _expand_ports(self, ports_list):
             if '-' not in i:
                 ports.append(int(i))
             else:
-                l, h = map(int, i.split('-'))
-                ports += range(l, h + 1)
+                l, h = list(map(int, i.split('-')))
+                ports += list(range(l, h + 1))
         return ports
 
     def _fuzzy_true(self, value):
@@ -62,7 +64,7 @@ def _fuzzy_false(self, value):
         return value.lower() in ['no', 'off', 'false', 'disable', 'disabled']
 
     def is_configured(self, opt):
-        return opt.lower() in self._dict.keys()
+        return opt.lower() in list(self._dict.keys())
 
     def is_unconfigured(self, opt):
         return not self.is_configured(opt)

fakenet/diverters/fnpacket.py

@@ -1,7 +1,9 @@
+# Copyright (C) 2016-2023 Mandiant, Inc. All rights reserved.
+
 import dpkt
 import socket
 import logging
-import debuglevels
+from . import debuglevels
 
 
 class PacketCtx(object):
@@ -64,7 +66,7 @@ def __init__(self, label, raw):
         self.dkey = None
 
         # Parse as much as possible
-        self.ipver = ((ord(self._raw[0]) & 0xf0) >> 4)
+        self.ipver = ((self._raw[0] & 0xf0) >> 4)
         if self.ipver == 4:
             self._parseIpv4()
         elif self.ipver == 6:

fakenet/diverters/linutil.py

@@ -1,3 +1,5 @@
+# Copyright (C) 2016-2023 Mandiant, Inc. All rights reserved.
+
 import os
 import re
 import glob
@@ -8,7 +10,7 @@
 import threading
 import subprocess
 import netfilterqueue
-from debuglevels import *
+from .debuglevels import *
 from collections import defaultdict
 from . import diverterbase
 
@@ -263,7 +265,7 @@ class LinUtilMixin(diverterbase.DiverterPerOSDelegate):
 
     def init_linux_mixin(self):
         self.old_dns = None
-        self.iptables_captured = ''
+        self.iptables_captured = b''
 
     def getNewDestinationIp(self, ip):
         """On Linux, FTP tests fail if IP redirection uses the external IP, so
@@ -294,18 +296,18 @@ def fix_dns(self):
         return False
 
     def linux_capture_iptables(self):
-        self.iptables_captured = ''
+        self.iptables_captured = b''
         ret = None
 
         try:
             p = subprocess.Popen(['iptables-save'], stdout=subprocess.PIPE)
             while True:
                 buf = p.stdout.read()
-                if buf == '':
+                if buf == b'':
                     break
                 self.iptables_captured += buf
 
-            if self.iptables_captured == '':
+            if self.iptables_captured == b'':
                 self.logger.warning('Null iptables-save output, likely not ' +
                                     'privileged')
             ret = p.wait()
@@ -397,7 +399,7 @@ def linux_get_next_nfqueue_numbers(self, n):
         existing_queues = self.linux_get_current_nfnlq_bindings()
 
         next_qnos = list()
-        for qno in xrange(QNO_MAX + 1):
+        for qno in range(QNO_MAX + 1):
             if qno not in existing_queues:
                 next_qnos.append(qno)
                 if len(next_qnos) == n:

fakenet/diverters/linux.py

@@ -1,3 +1,5 @@
+# Copyright (C) 2016-2023 Mandiant, Inc. All rights reserved.
+
 import sys
 import dpkt
 import time
@@ -7,10 +9,10 @@
 import threading
 import subprocess
 import netfilterqueue
-from linutil import *
+from .linutil import *
 from . import fnpacket
-from debuglevels import *
-from diverterbase import *
+from .debuglevels import *
+from .diverterbase import *
 from collections import namedtuple
 from netfilterqueue import NetfilterQueue