From f39588238526293db433a4da05ca48222578cc5e Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Tue, 11 Feb 2025 22:28:42 +0000 Subject: [PATCH 01/19] feat: AppArmor confinement for codejail-service Allow codejail-service to actually run code (and run it securely) by giving it the needed confinement. Previously it would run but refuse to execute code since it would detect the insecure environment; now, the startup safety checks pass and the code-exec endpoint works as expected. - Add an AppArmor profile with fairly strict rules. It needs to be thoroughly vetted and to have exceptions added before it can be used in production, but it's be fine for devstack. Some parts are based on the existing edxapp apparmor config without careful review. - Apply the profile to the codejail service in docker-compose. - Add Django configs for codejail service. - Add documentation for installing the profile so that it is available for use on the dev's machine. Also: - Add configuration and documentation for edxapp to actually call the codejail service, disabled by default. (Will later want to make this default to true, once the service is working properly.) - Update image name in docker-compose to follow rename in https://github.com/edx/public-dockerfiles/pull/102 Currently edxapp gets an error back from codejail-service, and then isn't able to read that error; separate work in the app repo will be needed to fix those. (The first issue relates to python_path, and the other to not returning globals_dict when there's an emsg.) But the integration is working otherwise. --- codejail.profile | 75 ++++++++++++++++++++++++++++++ docker-compose.yml | 4 +- docs/codejail.rst | 43 +++++++++++++++++ docs/index.rst | 1 + py_configuration_files/cms.py | 7 +++ py_configuration_files/codejail.py | 25 ++++++++++ py_configuration_files/lms.py | 6 +++ 7 files changed, 160 insertions(+), 1 deletion(-) create mode 100644 codejail.profile create mode 100644 docs/codejail.rst diff --git a/codejail.profile b/codejail.profile new file mode 100644 index 0000000000..9a47655d70 --- /dev/null +++ b/codejail.profile @@ -0,0 +1,75 @@ +# AppArmor profile for running codejail-service in devstack. +# +# #=========# +# # WARNING # +# #=========# +# +# This is not a complete and secure apparmor profile! Do not use this +# in any deployed environment (even a staging environment) without +# careful inspection and modification to fit your needs. +# +# Failure to apply a secure apparmor profile *will* likely result in a +# compromise of your environment by an attacker. +# +# We may at some point make this file good enough for confinement in +# production, but for now it is only intended to be used in devstack. + + +#include + +# Declare ABI version explicitly to ensure that confinement is +# actually applied appropriately on newer Ubuntu. +abi , + +# This outer profile applies to the entire container, and isn't as +# important. If the sandbox profile doesn't work, it's not likely that +# the outer one is going to help. But there may be some small value in +# defense-in-depth, as it's possible that a bug in the child (sandbox) +# profile isn't present in the outer one. +profile codejail_service flags=(attach_disconnected,mediate_deleted) { + #include + + # Filesystem access -- self-explanatory + file, + + # `network` is required for sudo + # TODO: Restrict this so that general network access is not permitted + network, + + # Various capabilities required for sudoing to sandbox (setuid, + # setgid, audit_write) and for sending a kill signal (kill). + capability setuid setgid audit_write kill, + + # Allow sending a kill signal to the sandbox when the execution + # runs beyond time limits. + signal (send) set=(kill) peer=codejail_service//child, + + # Allow executing this binary, but force a transition to the specified + # profile (and scrub the environment). + /sandbox/venv/bin/python Cx -> child, + + # This is the important apparmor profile -- the one that actually + # constrains the sandbox Python process. + profile child flags=(attach_disconnected,mediate_deleted) { + #include + + # Read and run binaries and libraries in the virtualenv. This + # includes the sandbox's copy of Python as well as any + # dependencies that have been installed for inclusion in + # sandboxes. + /sandbox/venv/** rm, + + # Codejail has a hardcoded reference to this file path, although the + # use of /tmp specifically may be controllable with environment variables: + # https://github.com/openedx/codejail/blob/0165d9ca351/codejail/util.py#L15 + /tmp/codejail-*/ r, + /tmp/codejail-*/** rw, + + # Allow interactive terminal during development + /dev/pts/* rw, + + # Allow receiving a kill signal from the webapp when the execution + # runs beyond time limits. + signal (receive) set=(kill) peer=codejail_service, + } +} diff --git a/docker-compose.yml b/docker-compose.yml index 4dd1598d5f..7ba7bca713 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -726,11 +726,13 @@ services: hostname: codejail.devstack.edx stdin_open: true tty: true - image: edxops/codejail-dev:latest + image: edxops/codejail-service-dev:latest environment: DJANGO_SETTINGS_MODULE: codejail_service.settings.devstack ports: - "18030:8080" + security_opt: + - apparmor=codejail_service xqueue: container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.xqueue" diff --git a/docs/codejail.rst b/docs/codejail.rst new file mode 100644 index 0000000000..2e3089b4d3 --- /dev/null +++ b/docs/codejail.rst @@ -0,0 +1,43 @@ +Codejail service +################ + +The ``codejail`` devstack component (codejail-service) requires some additional configuration before it can be enabled. This page describes how to set it up and debug it. + +Background +********** + +Both LMS and CMS can run Python code submitted by instructors and learners in order to implement custom Python-graded problems. By default this involves running the code on the same host as edxapp itself. Ordinarily this would be quite dangerous, but we use a sandboxing library called `codejail `__ in order to confine the code execution in terms of disk and network access as well as memory, CPU, and other resource limits. Part of these restrictions are implemented via AppArmor, a utility available in some Linux distributions (including Debian and Ubuntu). + +While AppArmor provides good protection, a sandbox escape could still be possible due to misconfiguration or bugs in AppArmor. For defense in depth, we're setting up a dedicated `codejail service `__ that will perform code execution for edxapp and which will allow further isolation. + +The default edxapp codejail defaults to unsafe, direct execution of Python code, and this remains true in devstack. We don't even have a way to run on-host codejail securely in devstack. In constrast, the codejail service refuses to run if codejail has not been configured properly, and we've included a way to run it in devstack. + +Configuration +************* + +In order to run the codejail devstack component: + +1. Install AppArmor: ``sudo apt install apparmor`` +2. Add the codejail AppArmor profile to your OS, or update it: ``sudo apparmor_parser --replace -W codejail.profile`` +3. Configure LMS and CMS to use the codejail-service by changing ``ENABLE_CODEJAIL_REST_SERVICE`` to ``True`` in ``py_configuration_files/{lms,cms}.py`` +4. Run ``make codejail-up`` + +The service does not need any provisioning, and does not have dependencies. + +Development +*********** + +Changes to the AppArmor profile must be coordinated with changes to the Dockerfile, as they need to agree on filesystem paths. + +Any time you update the profile, you'll need to re-run the command to apply the profile. + +The profile file contains the directive ``profile codejail_service``. That defines the name of the profile when it is installed into the kernel. In order to change that name, you must first remove the profile **under the old name**, then install a new profile under the new name. To remove a profile, use the ``--remove`` action instead of the ``-replace`` action: : ``sudo apparmor_parser --remove -W codejail.profile`` + +The profile name must also agree with the relevant ``security_opt`` line in devstack's ``docker-compose.yml``. + +Debugging +********* + +To check whether the profile has been applied, run ``sudo aa-status | grep codejail``. This won't tell you if the profile is out of date, but it will tell you if you have *some* version of it installed. + +If you need to debug the confinement, either because it is restricting too much or too little, a good strategy is to run ``tail -F /var/log/kern.log | grep codejail`` and watch for ``DENIED`` lines. You should expect to see several appear during service startup, as the service is designed to probe the confinement as part of its initial healthcheck. diff --git a/docs/index.rst b/docs/index.rst index a822ef5f61..2d2eb45cea 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -27,3 +27,4 @@ Contents troubleshoot_general_tips manual_upgrades advanced_configuration + codejail diff --git a/py_configuration_files/cms.py b/py_configuration_files/cms.py index 1200a61b06..504af61d62 100644 --- a/py_configuration_files/cms.py +++ b/py_configuration_files/cms.py @@ -312,6 +312,13 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing xblock_duplicated_event_setting = EVENT_BUS_PRODUCER_CONFIG['org.openedx.content_authoring.xblock.duplicated.v1'] xblock_duplicated_event_setting['course-authoring-xblock-lifecycle']['enabled'] = True +############################ Codejail ############################ + +# Disabled by default since codejail service needs to be configured +# and started separately. See docs/codejail.rst for details. +ENABLE_CODEJAIL_REST_SERVICE = False +CODE_JAIL_REST_SERVICE_HOST = "http://edx.devstack.codejail:8080" + ################# New settings must go ABOVE this line ################# ######################################################################## diff --git a/py_configuration_files/codejail.py b/py_configuration_files/codejail.py index 7e466d1bff..5eb0576340 100644 --- a/py_configuration_files/codejail.py +++ b/py_configuration_files/codejail.py @@ -1,3 +1,28 @@ """Settings for devstack use.""" from codejail_service.settings.local import * # pylint: disable=wildcard-import + +ALLOWED_HOSTS = [ + # When called from outside of docker's network (dev's terminal) + 'localhost', + # When called from another container (lms, cms) + 'edx.devstack.codejail', +] + +CODEJAIL_ENABLED = True + +CODE_JAIL = { + # These values are coordinated with the Dockerfile and AppArmor profile + 'python_bin': '/sandbox/venv/bin/python', + 'user': 'sandbox', + + # Configurable limits. + 'limits': { + # CPU-seconds + 'CPU': 3, + # 100 MiB memory + 'VMEM': 100 * 1024 * 1024, + # Clock seconds + 'REALTIME': 3, + }, +} diff --git a/py_configuration_files/lms.py b/py_configuration_files/lms.py index 9aacd6d266..2d4105057d 100644 --- a/py_configuration_files/lms.py +++ b/py_configuration_files/lms.py @@ -554,6 +554,12 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing 'http://localhost:1996', # frontend-app-learner-dashboard ] +############################ Codejail ############################ + +# Disabled by default since codejail service needs to be configured +# and started separately. See docs/codejail.rst for details. +ENABLE_CODEJAIL_REST_SERVICE = False +CODE_JAIL_REST_SERVICE_HOST = "http://edx.devstack.codejail:8080" ################# New settings must go ABOVE this line ################# ######################################################################## From baa60c91b71577b0a70f8968ac1378780ab4118c Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Thu, 13 Feb 2025 19:13:51 +0000 Subject: [PATCH 02/19] fixup! Switch to no-# include; add comments We can use either `#include` or `include` and it might be less confusing to use the one that doesn't look like a comment. I've added comments to some directives that I now understand better. --- codejail.profile | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/codejail.profile b/codejail.profile index 9a47655d70..8d0d377cb4 100644 --- a/codejail.profile +++ b/codejail.profile @@ -14,8 +14,9 @@ # We may at some point make this file good enough for confinement in # production, but for now it is only intended to be used in devstack. - -#include +# Sets standard variables used by abstractions/base, later. Controlled +# by OS, see /etc/apparmor.d/tunables/global for contents. +include # Declare ABI version explicitly to ensure that confinement is # actually applied appropriately on newer Ubuntu. @@ -27,7 +28,12 @@ abi , # defense-in-depth, as it's possible that a bug in the child (sandbox) # profile isn't present in the outer one. profile codejail_service flags=(attach_disconnected,mediate_deleted) { - #include + + # Allow access to a variety of commonly needed, generally safe things + # (such as reading /dev/random, free memory, etc.) + # + # Manpage: "Includes files that should be readable and writable in all profiles." + include # Filesystem access -- self-explanatory file, @@ -51,7 +57,11 @@ profile codejail_service flags=(attach_disconnected,mediate_deleted) { # This is the important apparmor profile -- the one that actually # constrains the sandbox Python process. profile child flags=(attach_disconnected,mediate_deleted) { - #include + + # This inner profile also gets general access to "safe" + # actions; we could list those explicitly out of caution but + # it could get pretty verbose. + include # Read and run binaries and libraries in the virtualenv. This # includes the sandbox's copy of Python as well as any From 672ca5278bf1665abd53f905568656818c7001eb Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Thu, 13 Feb 2025 23:27:51 +0000 Subject: [PATCH 03/19] fixup! Remove attach_disconnected and add more comments Just about every sample policy on the web includes attach_disconnected but the manpage describes it as a debugging tool that is not safe for general use. --- codejail.profile | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/codejail.profile b/codejail.profile index 8d0d377cb4..4b2c56af7e 100644 --- a/codejail.profile +++ b/codejail.profile @@ -27,7 +27,7 @@ abi , # the outer one is going to help. But there may be some small value in # defense-in-depth, as it's possible that a bug in the child (sandbox) # profile isn't present in the outer one. -profile codejail_service flags=(attach_disconnected,mediate_deleted) { +profile codejail_service flags=(mediate_deleted) { # Allow access to a variety of commonly needed, generally safe things # (such as reading /dev/random, free memory, etc.) @@ -50,13 +50,20 @@ profile codejail_service flags=(attach_disconnected,mediate_deleted) { # runs beyond time limits. signal (send) set=(kill) peer=codejail_service//child, - # Allow executing this binary, but force a transition to the specified - # profile (and scrub the environment). + # The core of the confinement: When the sandbox Python is executed, switch to + # the (extremely constrained) child profile. + # + # Manpage: "Cx: transition to subprofile on execute -- scrub the environment" /sandbox/venv/bin/python Cx -> child, # This is the important apparmor profile -- the one that actually # constrains the sandbox Python process. - profile child flags=(attach_disconnected,mediate_deleted) { + # + # mediate_deleted is not well documented, but it seems to indicate that + # apparmor will continue to make policy decisions in cases where a confined + # executable has a handle to a file's inode even after the file is removed + # from the filesystem. + profile child flags=(mediate_deleted) { # This inner profile also gets general access to "safe" # actions; we could list those explicitly out of caution but @@ -69,13 +76,17 @@ profile codejail_service flags=(attach_disconnected,mediate_deleted) { # sandboxes. /sandbox/venv/** rm, + # Allow access to the temporary directories that are set up by + # codejail, one for each code-exec call. Each /tmp/code-XXXXX + # contains one execution. + # # Codejail has a hardcoded reference to this file path, although the # use of /tmp specifically may be controllable with environment variables: # https://github.com/openedx/codejail/blob/0165d9ca351/codejail/util.py#L15 /tmp/codejail-*/ r, /tmp/codejail-*/** rw, - # Allow interactive terminal during development + # Allow interactive terminal in devstack. /dev/pts/* rw, # Allow receiving a kill signal from the webapp when the execution From 0f29e5cf72246db0d64f2adc53d40b1790696074 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Thu, 13 Feb 2025 23:39:51 +0000 Subject: [PATCH 04/19] fixup! add explanation for abi --- codejail.profile | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/codejail.profile b/codejail.profile index 4b2c56af7e..1ff7837748 100644 --- a/codejail.profile +++ b/codejail.profile @@ -18,8 +18,10 @@ # by OS, see /etc/apparmor.d/tunables/global for contents. include -# Declare ABI version explicitly to ensure that confinement is -# actually applied appropriately on newer Ubuntu. +# Require that the system understands the feature set that this policy was written +# for. If we didn't include this, then on Ubuntu >= 22.04, AppArmor might assume +# the wrong feature set was requested, and some rules might become too permissive. +# See https://github.com/netblue30/firejail/issues/3659#issuecomment-711074899 abi , # This outer profile applies to the entire container, and isn't as From 9d81e2a8057804c8eccb3c60a116d2a376d81673 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Thu, 13 Feb 2025 23:42:17 +0000 Subject: [PATCH 05/19] fixup! add note about subprofile path --- codejail.profile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/codejail.profile b/codejail.profile index 1ff7837748..f29b0bcfbc 100644 --- a/codejail.profile +++ b/codejail.profile @@ -55,6 +55,8 @@ profile codejail_service flags=(mediate_deleted) { # The core of the confinement: When the sandbox Python is executed, switch to # the (extremely constrained) child profile. # + # This path needs to be coordinated with the Dockerfile and Django settings. + # # Manpage: "Cx: transition to subprofile on execute -- scrub the environment" /sandbox/venv/bin/python Cx -> child, From edeeaeafb674d55219ac49862b8caa08a970f799 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Thu, 13 Feb 2025 23:44:05 +0000 Subject: [PATCH 06/19] fixup! Make it easier to enable (rather than copying the default) --- docs/codejail.rst | 2 +- py_configuration_files/cms.py | 3 ++- py_configuration_files/lms.py | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/codejail.rst b/docs/codejail.rst index 2e3089b4d3..96da514a5b 100644 --- a/docs/codejail.rst +++ b/docs/codejail.rst @@ -19,7 +19,7 @@ In order to run the codejail devstack component: 1. Install AppArmor: ``sudo apt install apparmor`` 2. Add the codejail AppArmor profile to your OS, or update it: ``sudo apparmor_parser --replace -W codejail.profile`` -3. Configure LMS and CMS to use the codejail-service by changing ``ENABLE_CODEJAIL_REST_SERVICE`` to ``True`` in ``py_configuration_files/{lms,cms}.py`` +3. Configure LMS and CMS to use the codejail-service by uncommenting ``# ENABLE_CODEJAIL_REST_SERVICE = True`` in ``py_configuration_files/{lms,cms}.py`` 4. Run ``make codejail-up`` The service does not need any provisioning, and does not have dependencies. diff --git a/py_configuration_files/cms.py b/py_configuration_files/cms.py index 504af61d62..3431d489fa 100644 --- a/py_configuration_files/cms.py +++ b/py_configuration_files/cms.py @@ -316,7 +316,8 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing # Disabled by default since codejail service needs to be configured # and started separately. See docs/codejail.rst for details. -ENABLE_CODEJAIL_REST_SERVICE = False + +# ENABLE_CODEJAIL_REST_SERVICE = True CODE_JAIL_REST_SERVICE_HOST = "http://edx.devstack.codejail:8080" diff --git a/py_configuration_files/lms.py b/py_configuration_files/lms.py index 2d4105057d..426f4388be 100644 --- a/py_configuration_files/lms.py +++ b/py_configuration_files/lms.py @@ -558,7 +558,8 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing # Disabled by default since codejail service needs to be configured # and started separately. See docs/codejail.rst for details. -ENABLE_CODEJAIL_REST_SERVICE = False + +# ENABLE_CODEJAIL_REST_SERVICE = True CODE_JAIL_REST_SERVICE_HOST = "http://edx.devstack.codejail:8080" ################# New settings must go ABOVE this line ################# From 260f966427a5fef4f36121597ffd34837013a719 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Fri, 14 Feb 2025 16:11:07 +0000 Subject: [PATCH 07/19] fixup! Deny outbound connections; clean up capabilities - Document what netlink is and why it's needed - Allow TCP connections, but not establishing outbound ones - Split apart capability lines into two blocks for easier commenting --- codejail.profile | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/codejail.profile b/codejail.profile index f29b0bcfbc..ceb91b7956 100644 --- a/codejail.profile +++ b/codejail.profile @@ -40,13 +40,18 @@ profile codejail_service flags=(mediate_deleted) { # Filesystem access -- self-explanatory file, - # `network` is required for sudo - # TODO: Restrict this so that general network access is not permitted - network, + # netlink is needed for sudo's interprocess communication + network netlink raw, - # Various capabilities required for sudoing to sandbox (setuid, - # setgid, audit_write) and for sending a kill signal (kill). - capability setuid setgid audit_write kill, + # Allow all of the various network operations required to listen, accept connection, etc. + network tcp, + # But then deny making a new *outbound* connection. + deny network (connect) tcp, + + # Required for sudoing to sandbox + capability setuid setgid audit_write, + # Allow sending a kill signal + capability kill, # Allow sending a kill signal to the sandbox when the execution # runs beyond time limits. From 058065e998f9412c8d7e2965effeb47bd036536c Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 15:46:28 +0000 Subject: [PATCH 08/19] fixup! Shrink docs for codejail setup (will move to service repo docs) --- docs/codejail.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/codejail.rst b/docs/codejail.rst index 96da514a5b..d86a7b7aca 100644 --- a/docs/codejail.rst +++ b/docs/codejail.rst @@ -6,11 +6,9 @@ The ``codejail`` devstack component (codejail-service) requires some additional Background ********** -Both LMS and CMS can run Python code submitted by instructors and learners in order to implement custom Python-graded problems. By default this involves running the code on the same host as edxapp itself. Ordinarily this would be quite dangerous, but we use a sandboxing library called `codejail `__ in order to confine the code execution in terms of disk and network access as well as memory, CPU, and other resource limits. Part of these restrictions are implemented via AppArmor, a utility available in some Linux distributions (including Debian and Ubuntu). +The `codejail-service `__ webapp is a wrapper around the `codejail `__ library. See the READMEs of each repo for more information on the special requirements for deploying codejail, in particular the AppArmor-based sandboxing. -While AppArmor provides good protection, a sandbox escape could still be possible due to misconfiguration or bugs in AppArmor. For defense in depth, we're setting up a dedicated `codejail service `__ that will perform code execution for edxapp and which will allow further isolation. - -The default edxapp codejail defaults to unsafe, direct execution of Python code, and this remains true in devstack. We don't even have a way to run on-host codejail securely in devstack. In constrast, the codejail service refuses to run if codejail has not been configured properly, and we've included a way to run it in devstack. +References to "codejail" can mean either the library or the service. In devstack, "codejail" usually refers to the service. Configuration ************* From 8e50c28e2e5fef37f9d71aa7be0d0811cdf880e5 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 15:56:02 +0000 Subject: [PATCH 09/19] fixup! Clarify instructions - `--add` vs `--replace` and when to use them - TODO around Mac - Hints that codejail.profile is a file provided in the repo --- docs/codejail.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/codejail.rst b/docs/codejail.rst index d86a7b7aca..58ece81cf5 100644 --- a/docs/codejail.rst +++ b/docs/codejail.rst @@ -13,21 +13,25 @@ References to "codejail" can mean either the library or the service. In devstack Configuration ************* +**TODO**: These instructions are for Linux. We need some instructions for how to do this on Mac; it probably involves accessing the Linux VM that docker is run inside of. + In order to run the codejail devstack component: 1. Install AppArmor: ``sudo apt install apparmor`` -2. Add the codejail AppArmor profile to your OS, or update it: ``sudo apparmor_parser --replace -W codejail.profile`` +2. Add the provided codejail AppArmor profile to your OS: ``sudo apparmor_parser --add -W ./codejail.profile`` 3. Configure LMS and CMS to use the codejail-service by uncommenting ``# ENABLE_CODEJAIL_REST_SERVICE = True`` in ``py_configuration_files/{lms,cms}.py`` 4. Run ``make codejail-up`` The service does not need any provisioning, and does not have dependencies. +Over time, the AppArmor profile may need to be updated. Changes to the file do not automatically cause changes to the version that has been installed in the OS. When significant changes have been made to the profile, you'll need to re-install the profile. This can be done by passing ``--replace`` instead of ``--add``, like so: ``sudo apparmor_parser --replace -W ./codejail.profile`` + Development *********** Changes to the AppArmor profile must be coordinated with changes to the Dockerfile, as they need to agree on filesystem paths. -Any time you update the profile, you'll need to re-run the command to apply the profile. +Any time you update the profile, you'll need to update the profile in your OS as well: ``sudo apparmor_parser --replace -W ./codejail.profile`` The profile file contains the directive ``profile codejail_service``. That defines the name of the profile when it is installed into the kernel. In order to change that name, you must first remove the profile **under the old name**, then install a new profile under the new name. To remove a profile, use the ``--remove`` action instead of the ``-replace`` action: : ``sudo apparmor_parser --remove -W codejail.profile`` From 72ac296ec5e180b82064e9dd28f5189ec1f60714 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 15:59:50 +0000 Subject: [PATCH 10/19] fixup! Clarify discussion of profile renaming --- docs/codejail.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/codejail.rst b/docs/codejail.rst index 58ece81cf5..83ec4924dd 100644 --- a/docs/codejail.rst +++ b/docs/codejail.rst @@ -33,9 +33,7 @@ Changes to the AppArmor profile must be coordinated with changes to the Dockerfi Any time you update the profile, you'll need to update the profile in your OS as well: ``sudo apparmor_parser --replace -W ./codejail.profile`` -The profile file contains the directive ``profile codejail_service``. That defines the name of the profile when it is installed into the kernel. In order to change that name, you must first remove the profile **under the old name**, then install a new profile under the new name. To remove a profile, use the ``--remove`` action instead of the ``-replace`` action: : ``sudo apparmor_parser --remove -W codejail.profile`` - -The profile name must also agree with the relevant ``security_opt`` line in devstack's ``docker-compose.yml``. +The profile file contains the directive ``profile codejail_service``. That defines the name of the profile when it is installed into the OS, and must agree with the relevant ``security_opt`` line in ``docker-compose.yml``. This name should not be changed, as it creates a confusing situation and would require every developer who uses codejail-service to do a number of manual steps. (Profiles can't be renamed *within* the OS; they must first be removed **under the old name**, and then a new profile must be installed under the new name.) Debugging ********* From 2171ad7ea4bcb0555cb2a33a7530814ba7176ed0 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 16:07:48 +0000 Subject: [PATCH 11/19] fixup! Use name of subprofile rather than a description of it --- codejail.profile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/codejail.profile b/codejail.profile index ceb91b7956..0a73f4e5c1 100644 --- a/codejail.profile +++ b/codejail.profile @@ -53,7 +53,7 @@ profile codejail_service flags=(mediate_deleted) { # Allow sending a kill signal capability kill, - # Allow sending a kill signal to the sandbox when the execution + # Allow sending a kill signal to the child subprofile when the execution # runs beyond time limits. signal (send) set=(kill) peer=codejail_service//child, From b27d15f04dd4709313ef5165763a8ec7c9bf9b53 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 16:09:11 +0000 Subject: [PATCH 12/19] fixup! Link to manpage --- codejail.profile | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/codejail.profile b/codejail.profile index 0a73f4e5c1..d0eca8e2f0 100644 --- a/codejail.profile +++ b/codejail.profile @@ -8,12 +8,17 @@ # in any deployed environment (even a staging environment) without # careful inspection and modification to fit your needs. # +# See https://manpages.ubuntu.com/manpages/noble/man5/apparmor.d.5.html +# or `man apparmor.d` for documentation of syntax and options. +# # Failure to apply a secure apparmor profile *will* likely result in a # compromise of your environment by an attacker. # # We may at some point make this file good enough for confinement in # production, but for now it is only intended to be used in devstack. + + # Sets standard variables used by abstractions/base, later. Controlled # by OS, see /etc/apparmor.d/tunables/global for contents. include From 5b07706495787a2d10cdc360bbc8609c4710ccce Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 16:18:46 +0000 Subject: [PATCH 13/19] fixup! Explain `m` access mode --- codejail.profile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/codejail.profile b/codejail.profile index d0eca8e2f0..99fcca6f64 100644 --- a/codejail.profile +++ b/codejail.profile @@ -88,6 +88,9 @@ profile codejail_service flags=(mediate_deleted) { # includes the sandbox's copy of Python as well as any # dependencies that have been installed for inclusion in # sandboxes. + # + # m: executable mapping, required for shared libraries used by some + # Python dependencies with C compontents, eg `nltk`. /sandbox/venv/** rm, # Allow access to the temporary directories that are set up by From e826482356b538ea9a9ee335b1f29b3ae67643a8 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 16:20:07 +0000 Subject: [PATCH 14/19] fixup! More explicit locations for sandbox path coordination --- py_configuration_files/codejail.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/py_configuration_files/codejail.py b/py_configuration_files/codejail.py index 5eb0576340..eb88400419 100644 --- a/py_configuration_files/codejail.py +++ b/py_configuration_files/codejail.py @@ -12,7 +12,8 @@ CODEJAIL_ENABLED = True CODE_JAIL = { - # These values are coordinated with the Dockerfile and AppArmor profile + # These values are coordinated with the Dockerfile (in edx/public-dockerfiles) + # and the AppArmor profile (codejail.profile in edx/devstack). 'python_bin': '/sandbox/venv/bin/python', 'user': 'sandbox', From f89e2ab38854c59539b05f8d6e55bbcfc7f83003 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 12:50:48 -0500 Subject: [PATCH 15/19] fixup! Apply suggestion re: Mac TODO Co-authored-by: Robert Raposa --- docs/codejail.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/codejail.rst b/docs/codejail.rst index 83ec4924dd..a33cdf4271 100644 --- a/docs/codejail.rst +++ b/docs/codejail.rst @@ -13,7 +13,7 @@ References to "codejail" can mean either the library or the service. In devstack Configuration ************* -**TODO**: These instructions are for Linux. We need some instructions for how to do this on Mac; it probably involves accessing the Linux VM that docker is run inside of. +These instructions are for Linux only. Additional research would be required to create instructions for a Mac, which probably involves accessing the Linux VM that docker is run inside of. In order to run the codejail devstack component: From f9f3eeca27c22831e85699d0d43d3f7fd084f549 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 12:52:12 -0500 Subject: [PATCH 16/19] fixup! Add "file" for clarity Co-authored-by: Robert Raposa --- docs/codejail.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/codejail.rst b/docs/codejail.rst index a33cdf4271..f9cba9767e 100644 --- a/docs/codejail.rst +++ b/docs/codejail.rst @@ -31,7 +31,7 @@ Development Changes to the AppArmor profile must be coordinated with changes to the Dockerfile, as they need to agree on filesystem paths. -Any time you update the profile, you'll need to update the profile in your OS as well: ``sudo apparmor_parser --replace -W ./codejail.profile`` +Any time you update the profile file, you'll need to update the profile in your OS as well: ``sudo apparmor_parser --replace -W ./codejail.profile`` The profile file contains the directive ``profile codejail_service``. That defines the name of the profile when it is installed into the OS, and must agree with the relevant ``security_opt`` line in ``docker-compose.yml``. This name should not be changed, as it creates a confusing situation and would require every developer who uses codejail-service to do a number of manual steps. (Profiles can't be renamed *within* the OS; they must first be removed **under the old name**, and then a new profile must be installed under the new name.) From f0e80ee517edaa9da67d9243bf4d72c66bf7e539 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 17:43:36 +0000 Subject: [PATCH 17/19] fixup! Comment on port mismatch --- py_configuration_files/cms.py | 4 +++- py_configuration_files/lms.py | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/py_configuration_files/cms.py b/py_configuration_files/cms.py index 3431d489fa..d2ef80c4d2 100644 --- a/py_configuration_files/cms.py +++ b/py_configuration_files/cms.py @@ -316,8 +316,10 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing # Disabled by default since codejail service needs to be configured # and started separately. See docs/codejail.rst for details. +#ENABLE_CODEJAIL_REST_SERVICE = True -# ENABLE_CODEJAIL_REST_SERVICE = True +# Note that this is exposed on port 8080 to other devstack services, +# but 18030 outside of Docker. CODE_JAIL_REST_SERVICE_HOST = "http://edx.devstack.codejail:8080" diff --git a/py_configuration_files/lms.py b/py_configuration_files/lms.py index 426f4388be..27d1625bcf 100644 --- a/py_configuration_files/lms.py +++ b/py_configuration_files/lms.py @@ -558,8 +558,10 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing # Disabled by default since codejail service needs to be configured # and started separately. See docs/codejail.rst for details. +#ENABLE_CODEJAIL_REST_SERVICE = True -# ENABLE_CODEJAIL_REST_SERVICE = True +# Note that this is exposed on port 8080 to other devstack services, +# but 18030 outside of Docker. CODE_JAIL_REST_SERVICE_HOST = "http://edx.devstack.codejail:8080" ################# New settings must go ABOVE this line ################# From a77c26cf84d8bb2bb55e7b8a42503e7fa34f3cf8 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 17:49:07 +0000 Subject: [PATCH 18/19] fixup! Rename child profile to codejail_sandbox Just a better (more descriptive) name, and has parallels to codejail_service profile name. --- codejail.profile | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/codejail.profile b/codejail.profile index 99fcca6f64..37d51cb2db 100644 --- a/codejail.profile +++ b/codejail.profile @@ -30,9 +30,9 @@ include abi , # This outer profile applies to the entire container, and isn't as -# important. If the sandbox profile doesn't work, it's not likely that +# important. If the inner profile doesn't work, it's not likely that # the outer one is going to help. But there may be some small value in -# defense-in-depth, as it's possible that a bug in the child (sandbox) +# defense-in-depth, as it's possible that a bug in the codejail_sandbox (inner) # profile isn't present in the outer one. profile codejail_service flags=(mediate_deleted) { @@ -58,17 +58,17 @@ profile codejail_service flags=(mediate_deleted) { # Allow sending a kill signal capability kill, - # Allow sending a kill signal to the child subprofile when the execution + # Allow sending a kill signal to the codejail_sandbox subprofile when the execution # runs beyond time limits. - signal (send) set=(kill) peer=codejail_service//child, + signal (send) set=(kill) peer=codejail_service//codejail_sandbox, # The core of the confinement: When the sandbox Python is executed, switch to - # the (extremely constrained) child profile. + # the (extremely constrained) codejail_sandbox profile. # # This path needs to be coordinated with the Dockerfile and Django settings. # # Manpage: "Cx: transition to subprofile on execute -- scrub the environment" - /sandbox/venv/bin/python Cx -> child, + /sandbox/venv/bin/python Cx -> codejail_sandbox, # This is the important apparmor profile -- the one that actually # constrains the sandbox Python process. @@ -77,7 +77,7 @@ profile codejail_service flags=(mediate_deleted) { # apparmor will continue to make policy decisions in cases where a confined # executable has a handle to a file's inode even after the file is removed # from the filesystem. - profile child flags=(mediate_deleted) { + profile codejail_sandbox flags=(mediate_deleted) { # This inner profile also gets general access to "safe" # actions; we could list those explicitly out of caution but From 1adda1895f961032c24c8a21b6a2f9ddcc384079 Mon Sep 17 00:00:00 2001 From: Tim McCormack Date: Wed, 19 Feb 2025 17:56:30 +0000 Subject: [PATCH 19/19] fixup! Clarify what "important" is relative to --- codejail.profile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/codejail.profile b/codejail.profile index 37d51cb2db..dbb4b4e983 100644 --- a/codejail.profile +++ b/codejail.profile @@ -30,7 +30,7 @@ include abi , # This outer profile applies to the entire container, and isn't as -# important. If the inner profile doesn't work, it's not likely that +# important as the inner (codejail_sandbox) profile. If the inner profile doesn't work, it's not likely that # the outer one is going to help. But there may be some small value in # defense-in-depth, as it's possible that a bug in the codejail_sandbox (inner) # profile isn't present in the outer one.