Merge pull request #1 from Ostorlab/feature/setup-agent

Setup Agent Nebula

README.md

@@ -1,35 +1,74 @@
-# Ostorlab Template Agent
+<h1 align="center">Agent Nebula</h1>
 
-This repo is a template for building an [Ostorlab](https://github.com/ostorlab/ostorlab) agent in Python. It ships with good best practices like:
+<p align="center">
+<img src="https://img.shields.io/badge/License-Apache_2.0-brightgreen.svg">
+<img src="https://img.shields.io/github/languages/top/ostorlab/agent_nebula">
+<img src="https://img.shields.io/github/stars/ostorlab/agent_nebula">
+<img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg">
+</p>
 
-* Github actions workflow
-* Linting checks with Ruff
-* Static typing checks with Mypy
-* Running the unit test with Pytest
+_The Nebula Agent is responsible for persisting all types of messages locally._
+
+---
+
+<p align="center">
+<img src="https://github.com/ostorlab/agent_nebula/blob/main/images/logo.png" alt="agent-nebula" />
+</p>
 
 ## Getting Started
+To perform your first scan, simply run the following command:
+```shell
+ostorlab scan run --install --agent agent/ostorlab/nebula link --url www.yourdomain.com --method GET
+``` 
+
+This command will download and install `agent/ostorlab/nebula` and target `www.yourdomain.com` with the `GET` method.
+For more information, please refer to the [Ostorlab Documentation](https://github.com/Ostorlab/ostorlab/blob/main/README.md)
+
+
+## Usage
+
+Agent Nebula can be installed directly from the ostorlab agent store or built from this repository.
+
+ ### Install directly from ostorlab agent store
+
+ ```shell
+ ostorlab agent install agent/ostorlab/nebula
+ ```
+
+You can then run the agent with the following command:
+```shell
+ostorlab scan run --agent agent/ostorlab/nebula link --url www.yourdomain.com --method GET
+```
+
+
+### Build directly from the repository
+
+ 1. To build nebula agent you need to have [ostorlab](https://pypi.org/project/ostorlab/) installed in your machine. If you have already installed ostorlab, you can skip this step.
+
+```shell
+pip3 install ostorlab
+```
+
+ 2. Clone this repository.
+
+```shell
+git clone https://github.com/Ostorlab/agent_nebula.git && cd agent_nebula
+```
+
+ 3. Build the agent image using ostorlab cli.
+
+ ```shell
+ ostorlab agent build --file=ostorlab.yaml
+ ```
+
+ You can pass the optional flag `--organization` to specify your organisation. The organization is empty by default.
 
-Here are links to good resources to get started:
-
-* [Write your first Agent](https://docs.ostorlab.co/tutorials/write-an-ostorlab-agent.html)
-* [Use Ostorlab to run scans](https://docs.ostorlab.co/tutorials/run-your-first-scan.html)
-* [Debugging and Testing Agents](https://docs.ostorlab.co/tutorials/debugging-agents.html)
-* [Understand Ostorlab Internals](https://docs.ostorlab.co/tutorials/life-of-a-scan.html)
-
-## Ideas for Agents to build
-
-Implementation of popular tools like:
-
-* ~~[semgrep](https://github.com/returntocorp/semgrep) for source code scanning.~~
-* [nbtscan](http://www.unixwiz.net/tools/nbtscan.html): Scans for open NETBIOS nameservers on your target’s network.
-* [onesixtyone](https://github.com/trailofbits/onesixtyone): Fast scanner to find publicly exposed SNMP services.
-* [Retire.js](http://retirejs.github.io/retire.js/): Scanner detecting the use of JavaScript libraries with known
-  vulnerabilities.
-* [snallygaster](https://github.com/hannob/snallygaster): Finds file leaks and other security problems on HTTP servers.
-* [testssl.sh](https://testssl.sh/): Identify various TLS/SSL weaknesses, including Heartbleed, CRIME and ROBOT.
-* ~~[TruffleHog](https://github.com/trufflesecurity/truffleHog): Searches through git repositories for high entropy
-  strings and secrets, digging deep into commit history.~~
-* [cve-bin-tool](https://github.com/intel/cve-bin-tool): Scan binaries for vulnerable components.
-* [XSStrike](https://github.com/s0md3v/XSStrike): XSS web vulnerability scanner with generative payload.
-* ~~[Subjack](https://github.com/haccer/subjack): Subdomain takeover scanning tool.~~
-* [DnsReaper](https://github.com/punk-security/dnsReaper): Subdomain takeover scanning tool.
+ 4. Run the agent using on of the following commands:
+	 * If you did not specify an organization when building the image:
+    ```shell
+    ostorlab scan run --agent agent//nebula link --url www.yourdomain.com --method GET
+    ```
+	 * If you specified an organization when building the image:
+    ```shell
+    ostorlab scan run --agent agent/[ORGANIZATION]/nebula link --url www.yourdomain.com --method GET
+    ```

agent/nebula_agent.py

@@ -0,0 +1,49 @@
+"""Nebula agent: responsible for persisting all types of messages."""
+
+import logging
+
+from ostorlab.agent import agent, definitions as agent_definitions
+from ostorlab.agent.message import message as m
+from ostorlab.runtimes import definitions as runtime_definitions
+from rich import logging as rich_logging
+
+logging.basicConfig(
+    format="%(message)s",
+    datefmt="[%X]",
+    level="INFO",
+    force=True,
+    handlers=[rich_logging.RichHandler(rich_tracebacks=True)],
+)
+logger = logging.getLogger(__name__)
+
+
+class NebulaAgent(agent.Agent):
+    """Agent responsible for persisting all types of messages to file type specified in the agent definition."""
+
+    def __init__(
+        self,
+        agent_definition: agent_definitions.AgentDefinition,
+        agent_settings: runtime_definitions.AgentSettings,
+    ) -> None:
+        super().__init__(agent_definition, agent_settings)
+        self._file_type = self.args.get("file_type")
+        self._file_path = self.args.get("file_path")
+
+    def process(self, message: m.Message) -> None:
+        """Process the message and persist it to the file type and location specified in the agent definition.
+
+        Args:
+            message: The message to process.
+        """
+        logger.info("processing message of selector : %s", message.selector)
+        # TODO (elyousfi5): add the logic to persist the message to the file type and location specified in the agent
+        logger.info(
+            "message persisted to file type: %s at location: %s",
+            self._file_type,
+            self._file_path,
+        )
+
+
+if __name__ == "__main__":
+    logger.info("starting agent ...")
+    NebulaAgent.main()

ostorlab.yaml

@@ -1,9 +1,78 @@
 kind: Agent
-name: template_agent # Agent name, must be unique by organisation to be published on the store.
-version: 0.0.0 # Must respect semantic versioning.
-description: Agent description. # Support for Markdown format.
-in_selectors: # List of input selectors, this is basically the list of messages the agent should receive.
-  - v3.healthcheck.ping
-out_selectors: [] # List of output selectors.
-docker_file_path : Dockerfile # Dockerfile path for automated releases.
-docker_build_root : . # Docker build dir for automated release build.
+name: nebula
+version: 0.0.1
+description: |
+  _The Nebula Agent is responsible for persisting all types of messages locally._
+
+  ## Getting Started
+  To perform your first scan, simply run the following command:
+  ```shell
+  ostorlab scan run --install --agent agent/ostorlab/nebula link --url www.yourdomain.com --method GET
+  ``` 
+
+  This command will download and install `agent/ostorlab/nebula` and target `www.yourdomain.com` with the `GET` method.
+  For more information, please refer to the [Ostorlab Documentation](https://github.com/Ostorlab/ostorlab/blob/main/README.md)
+
+
+  ## Usage
+
+  Agent Nebula can be installed directly from the ostorlab agent store or built from this repository.
+
+   ### Install directly from ostorlab agent store
+
+   ```shell
+   ostorlab agent install agent/ostorlab/nebula
+   ```
+
+  You can then run the agent with the following command:
+  ```shell
+  ostorlab scan run --agent agent/ostorlab/nebula link --url www.yourdomain.com --method GET
+  ```
+
+
+  ### Build directly from the repository
+
+   1. To build nebula agent you need to have [ostorlab](https://pypi.org/project/ostorlab/) installed in your machine. If you have already installed ostorlab, you can skip this step.
+
+  ```shell
+  pip3 install ostorlab
+  ```
+
+   2. Clone this repository.
+
+  ```shell
+  git clone https://github.com/Ostorlab/agent_nebula.git && cd agent_nebula
+  ```
+
+   3. Build the agent image using ostorlab cli.
+
+   ```shell
+   ostorlab agent build --file=ostorlab.yaml
+   ```
+
+   You can pass the optional flag `--organization` to specify your organisation. The organization is empty by default.
+
+   4. Run the agent using on of the following commands:
+  	 * If you did not specify an organization when building the image:
+      ```shell
+      ostorlab scan run --agent agent//nebula ip 8.8.8.8
+      ```
+  	 * If you specified an organization when building the image:
+      ```shell
+      ostorlab scan run --agent agent/[ORGANIZATION]/nebula link --url www.yourdomain.com --method GET
+      ```
+in_selectors:
+  - v3.asset.ip.v4
+  - v3.asset.ip.v6
+  - v3.asset.domain_name
+  - v3.asset.link
+out_selectors: []
+docker_file_path : Dockerfile
+docker_build_root : .
+args:
+  - name: "file_type"
+    type: "string"
+    description: "The type of the file where the message will be persisted."
+  - name: "file_path"
+    type: "string"
+    description: "The path of the file where the message will be persisted."

tests/conftest.py

@@ -1,10 +1 @@
-"""
-    Dummy conftest.py for template_agent.
-
-    If you don't know what this is for, just leave it empty.
-    Read more about conftest.py under:
-    - https://docs.pytest.org/en/stable/fixture.html
-    - https://docs.pytest.org/en/stable/writing_plugins.html
-"""
-
-# import pytest
+"""Pytest fixtures for Agent Nebula."""

tests/nebula_agent_test.py

@@ -0,0 +1,9 @@
+"""Unit tests for Nebula agent."""
+
+
+# TODO (elyousfi5): add tests for the Nebula agent
+
+
+def testNebulaAgent_always_persistMessages() -> None:
+    """Test Nebula agent."""
+    pass