diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..271c0fc2 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,76 @@ +# Repository Guidelines + +## Project Structure & Module Organization +This repository hosts Shuffle app implementations. Each app lives in a top-level folder named after the integration (e.g., `aws-ec2/`), and each release is versioned under a subfolder like `1.0.0/`. A typical app version contains: + +- `src/app.py`: the Shuffle SDK entry point. +- `api.yaml`: OpenAPI definition used by Shuffle. +- `requirements.txt`: Python dependencies for the app. +- `Dockerfile`: container build instructions for the app. +- `README.md`: app-specific usage and action documentation. +- Optional assets such as screenshots (`*.png`). + +In `api.yaml`, prefer an `authentication` block for shared credentials (URL, tokens, keys). Actions should only include auth parameters when they truly differ per call. + +## Build, Test, and Development Commands +Apps are built and run container-first via the Shuffle SDK image. From an app version directory: + +- `docker build -t shuffle-: .`: build the app image. +- `docker run --rm shuffle-:`: run the app container locally. + +For quick iteration on code, you can also run the Python entrypoint in a virtualenv: + +- `pip install -r requirements.txt` +- `python src/app.py --log-level DEBUG` + +## Coding Style & Naming Conventions +Use 4-space indentation and standard Python style. Keep functions `snake_case`, classes `CamelCase`, and constants `UPPER_SNAKE_CASE`. Match existing patterns in `src/app.py` and keep action names aligned with `api.yaml`. + +## Creating New Shuffle Apps (Agent Workflow) +Use an existing app as a template (e.g., `http/1.4.0/` or `aws-ec2/1.0.0/`) and follow the same folder layout. A minimal, working app version should include: + +- `api.yaml`: action definitions, parameters, and examples. +- `src/app.py`: class extending the Shuffle SDK (`shuffle_sdk.AppBase`). +- `requirements.txt`: third-party dependencies. +- `Dockerfile`: built on `frikky/shuffle:app_sdk`. + +When adding actions, ensure the `api.yaml` action name matches the method name in `src/app.py` and parameter names align exactly. Keep input parsing defensive (strings vs JSON), and return JSON-serializable results. For HTTP integrations, centralize auth and base URL handling and add a TLS `verify` option. If a service requires special payloads (e.g., ADF for Jira), accept JSON strings and pass through unchanged. Keep `api.yaml` examples realistic because they show up in the Shuffle UI. + +## Authentication & App Configuration +Most apps declare credentials in `api.yaml` under `authentication:` so Shuffle injects them automatically. In code, read those values as normal action arguments (Shuffle passes them into each action). Prefer a single auth helper in `src/app.py` (e.g., `_auth()` for tokens, `_build_api_base()` for base URLs) and reuse it across actions. If an integration supports multiple auth modes (token vs password), accept both and choose the provided one. + +Prefer small, focused actions (create, update, list, search) and document auth requirements and examples in the app `README.md`. + +## Manual Python App Notes (From Shuffle Docs) +- **SDK image choices:** Shuffle provides Alpine (slim), Kali (security tooling), and Blackarch (kitchen‑sink). This repo’s Dockerfiles typically use `frikky/shuffle:app_sdk` (Alpine‑based) unless a toolset requires otherwise. +- **Directory layout:** `api.yaml`, `Dockerfile`, `requirements.txt`, `README.md`, and `src/app.py` are expected in each app version. Complex apps can add additional modules under `src/` and import them from `app.py`. +- **Actions & wiring:** Every action in `api.yaml` must map to a method in `src/app.py` with the same name and argument list. Authentication parameters are passed into each action automatically when declared under `authentication:`. +- **Utility helpers:** In `AppBase`, you can use `self.get_file`, `self.set_files`, `self.update_file`, and cache helpers `self.get_cache`, `self.set_cache`, `self.delete_cache` for file and key/value workflows. +- **Prototyping:** Build and test your Python logic locally first, then wire it into `src/app.py`. Keep return values JSON‑serializable so Shuffle can consume them. +- **Upload & hotload:** After a prototype works, upload it to Shuffle (cloud) or hotload locally (on‑prem) by rebuilding the app image. Local Docker rebuilds are faster for iteration. + +## Testing, Hotloading, and CI/CD +- **Cloud upload test:** Use the Upload App API to add the app to your org, then run a workflow to validate actions. +- **Local hotload (on‑prem):** Place the app folder in `shuffle-apps/`, set `SHUFFLE_APP_HOTLOAD_FOLDER=./shuffle-apps`, then use the hot reload button in the UI. Allow ~20 seconds for the reload to complete. +- **Update workflow deps:** If you update an existing app version, remove and re‑add the app in any workflows that reference it. +- **Fast local iteration:** After the first upload, rebuild locally: `docker images | grep ` then `docker build . -t `. +- **CI/CD pattern:** Create a test workflow, upload a test app version, run the workflow via API, and validate `workflowexecution.workflow.validation.valid` before promoting. + +## Publishing Apps +- **OpenAPI apps:** Upload to your instance, then use the `/apps` page to publish so it appears on `shuffler.io`. +- **Python apps:** Fork `https://github.com/frikky/shuffle-apps`, add your app, and open a pull request to upstream. + +## Testing Guidelines +There is no repository-wide test suite. If you add tests for a specific app, keep them alongside the app version (e.g., `aws-ec2/1.0.0/tests/`) and document how to run them in that app’s `README.md`. + +## Commit & Pull Request Guidelines +Commit messages are short and descriptive, sometimes using a prefix like `fix:`. Follow that style and keep commits scoped to a single app/version when possible. + +For pull requests, include: + +- A clear description of the change and impacted app/version path. +- Updated `README.md` or `api.yaml` when behavior changes. +- Screenshots/assets if user-facing output or UI-related docs are affected. + +## Security & Configuration Tips +Many apps require API keys or credentials. Do not commit secrets; use environment variables or Shuffle configuration fields instead, and document required inputs in the app’s `README.md`. diff --git a/shuffle-tools/1.2.0/api.yaml b/shuffle-tools/1.2.0/api.yaml index ebbf7c3e..ab5337ee 100644 --- a/shuffle-tools/1.2.0/api.yaml +++ b/shuffle-tools/1.2.0/api.yaml @@ -1279,6 +1279,20 @@ actions: example: "tickets" schema: type: string + - name: return_values + description: Whether to include the cache values in the response + required: false + options: + - true + - false + multiline: false + example: "false" + value: "true" + schema: + type: string + returns: + schema: + type: string - name: get_cache_value description: Get a value saved to your organization in Shuffle. Deprecated for "get_datastore_value" parameters: @@ -1300,6 +1314,7 @@ actions: schema: type: string - name: delete_cache_value + description: Delete a value saved to your organization in Shuffle. Deprecated for "delete_datastore_value" parameters: - name: key @@ -1381,6 +1396,17 @@ actions: example: "tickets" schema: type: string + - name: return_values + description: Whether to include the cache values in the response + required: false + options: + - true + - false + multiline: false + example: "false" + value: "true" + schema: + type: string #- name: parse_ioc_new # description: Parse IOC's based on https://github.com/fhightower/ioc-finder # parameters: diff --git a/shuffle-tools/1.2.0/src/app.py b/shuffle-tools/1.2.0/src/app.py index 85275cf2..e1bbdac3 100644 --- a/shuffle-tools/1.2.0/src/app.py +++ b/shuffle-tools/1.2.0/src/app.py @@ -41,15 +41,16 @@ import concurrent.futures import multiprocessing -#from walkoff_app_sdk.app_base import AppBase +# from walkoff_app_sdk.app_base import AppBase from shuffle_sdk import AppBase -# Override exit(), sys.exit, and os._exit +# Override exit(), sys.exit, and os._exit # sys.exit() can be caught, meaning we can have a custom handler for it builtins.exit = sys.exit os.exit = sys.exit os._exit = sys.exit + class Tools(AppBase): __version__ = "1.2.0" app_name = ( @@ -86,7 +87,7 @@ def base64_conversion(self, string, operation): # Decode the base64 into an image and upload it as a file decoded_bytes = base64.b64decode(string) - # Make the bytes into unicode escaped bytes + # Make the bytes into unicode escaped bytes # UnicodeDecodeError - 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte try: decoded_bytes = str(decoded_bytes, "utf-8") @@ -96,7 +97,7 @@ def base64_conversion(self, string, operation): filename = "base64_image.png" file = { "filename": filename, - "data": decoded_bytes, + "data": decoded_bytes, } fileret = self.set_files([file]) @@ -107,7 +108,6 @@ def base64_conversion(self, string, operation): return value elif operation == "decode": - if "-" in string: string = string.replace("-", "+", -1) @@ -118,18 +118,19 @@ def base64_conversion(self, string, operation): if len(string) % 4 != 0: string += "=" * (4 - len(string) % 4) - # For loop this. It's stupid. - decoded_bytes = "" + decoded_bytes = "" try: decoded_bytes = base64.b64decode(string) except Exception as e: - return json.dumps({ - "success": False, - "reason": "Invalid Base64 - %s" % e, - }) + return json.dumps( + { + "success": False, + "reason": "Invalid Base64 - %s" % e, + } + ) - #if "incorrect padding" in str(e).lower(): + # if "incorrect padding" in str(e).lower(): # try: # decoded_bytes = base64.b64decode(string + "=") # except Exception as e: @@ -144,7 +145,6 @@ def base64_conversion(self, string, operation): # if "incorrect padding" in str(e).lower(): # return "Invalid Base64" - try: decoded_bytes = str(decoded_bytes, "utf-8") except: @@ -199,7 +199,6 @@ def send_sms_shuffle(self, apikey, phone_numbers, body): def send_email_shuffle(self, apikey, recipients, subject, body, attachments=""): recipients = self.parse_list_internal(recipients) - targets = [recipients] if ", " in recipients: targets = recipients.split(", ") @@ -207,9 +206,9 @@ def send_email_shuffle(self, apikey, recipients, subject, body, attachments=""): targets = recipients.split(",") data = { - "targets": targets, - "subject": subject, - "body": body, + "targets": targets, + "subject": subject, + "body": body, "type": "alert", "email_app": True, } @@ -222,11 +221,10 @@ def send_email_shuffle(self, apikey, recipients, subject, body, attachments=""): for item in attachments: new_file = self.get_file(file_ids) files.append(new_file) - + data["attachments"] = files except Exception as e: pass - url = "https://shuffler.io/functions/sendmail" headers = {"Authorization": "Bearer %s" % apikey} @@ -249,7 +247,8 @@ def dedup_and_merge(self, key, value, timeout, set_skipped=True): response = { "success": False, "datastore_key": cachekey, - "info": "All keys from the last %d seconds with the key '%s' have been merged. The result was set to SKIPPED in all other actions." % (timeout, key), + "info": "All keys from the last %d seconds with the key '%s' have been merged. The result was set to SKIPPED in all other actions." + % (timeout, key), "timeout": timeout, "original_value": value, "all_values": [], @@ -272,7 +271,7 @@ def dedup_and_merge(self, key, value, timeout, set_skipped=True): found_cache["value"].append(value) if "created" in found_cache: if found_cache["created"] + timeout + 3 < time.time(): - set_skipped = False + set_skipped = False response["success"] = True response["all_values"] = found_cache["value"] @@ -280,24 +279,37 @@ def dedup_and_merge(self, key, value, timeout, set_skipped=True): return json.dumps(response) else: - self.logger.info("Dedup-key is already handled in another workflow with timeout %d" % timeout) + self.logger.info( + "Dedup-key is already handled in another workflow with timeout %d" + % timeout + ) self.set_cache(cachekey, json.dumps(found_cache["value"])) if set_skipped == True: self.action_result["status"] = "SKIPPED" - self.action_result["result"] = json.dumps({ - "status": False, - "reason": "Dedup-key is already handled in another workflow with timeout %d" % timeout, - }) + self.action_result["result"] = json.dumps( + { + "status": False, + "reason": "Dedup-key is already handled in another workflow with timeout %d" + % timeout, + } + ) - self.send_result(self.action_result, {"Authorization": "Bearer %s" % self.authorization}, "/api/v1/streams") + self.send_result( + self.action_result, + {"Authorization": "Bearer %s" % self.authorization}, + "/api/v1/streams", + ) return found_cache parsedvalue = [value] resp = self.set_cache(cachekey, json.dumps(parsedvalue)) - self.logger.info("Sleeping for %d seconds while waiting for cache to fill up elsewhere" % timeout) + self.logger.info( + "Sleeping for %d seconds while waiting for cache to fill up elsewhere" + % timeout + ) time.sleep(timeout) found_cache = self.get_cache(cachekey) @@ -307,7 +319,6 @@ def dedup_and_merge(self, key, value, timeout, set_skipped=True): self.delete_cache(cachekey) return json.dumps(response) - # https://github.com/fhightower/ioc-finder def parse_file_ioc(self, file_ids, input_type="all"): def parse(data): @@ -326,7 +337,8 @@ def parse(data): for subitem in subvalue: data = { "data": subitem, - "data_type": "%s_%s" % (key[:-1], subkey), + "data_type": "%s_%s" + % (key[:-1], subkey), } if data not in newarray: newarray.append(data) @@ -405,10 +417,7 @@ def set_json_key(self, json_object, key, value): try: json_object = json.loads(json_object) except json.decoder.JSONDecodeError as e: - return { - "success": False, - "reason": "Item is not valid JSON" - } + return {"success": False, "reason": "Item is not valid JSON"} if isinstance(json_object, list): if len(json_object) == 1: @@ -416,16 +425,15 @@ def set_json_key(self, json_object, key, value): else: return { "success": False, - "reason": "Item is valid JSON, but can't handle lists. Use .#" + "reason": "Item is valid JSON, but can't handle lists. Use .#", } - #if not isinstance(json_object, object): + # if not isinstance(json_object, object): # return { # "success": False, # "reason": "Item is not valid JSON (2)" # } - if isinstance(value, str): try: value = json.loads(value) @@ -435,7 +443,7 @@ def set_json_key(self, json_object, key, value): # Handle JSON paths if "." in key: base_object = json.loads(json.dumps(json_object)) - #base_object.output.recipients.notificationEndpointIds = ... + # base_object.output.recipients.notificationEndpointIds = ... keys = key.split(".") if len(keys) >= 1: @@ -444,14 +452,14 @@ def set_json_key(self, json_object, key, value): # This is awful :) buildstring = "base_object" for subkey in keys: - buildstring += f"[\"{subkey}\"]" + buildstring += f'["{subkey}"]' buildstring += f" = {value}" - #output = + # output = exec(buildstring) json_object = base_object - #json_object[first_object] = base_object + # json_object[first_object] = base_object else: json_object[key] = value @@ -520,9 +528,9 @@ def replace_value_from_dictionary(self, input_data, mapping, default_value=""): except: self.logger.info(f"Failed mapping output data for key {key}") - return input_data + return input_data - # Changed with 1.1.0 to run with different returns + # Changed with 1.1.0 to run with different returns def regex_capture_group(self, input_data, regex): try: returnvalues = { @@ -533,8 +541,8 @@ def regex_capture_group(self, input_data, regex): found = False for item in matches: if isinstance(item, str): - found = True - name = "group_0" + found = True + name = "group_0" try: returnvalues[name].append(item) except: @@ -542,7 +550,7 @@ def regex_capture_group(self, input_data, regex): else: for i in range(0, len(item)): - found = True + found = True name = "group_%d" % i try: returnvalues[name].append(item[i]) @@ -558,10 +566,7 @@ def regex_capture_group(self, input_data, regex): "reason": "Bad regex pattern: %s" % e, } - def regex_replace( - self, input_data, regex, replace_string="", ignore_case="False" - ): - + def regex_replace(self, input_data, regex, replace_string="", ignore_case="False"): if ignore_case.lower().strip() == "true": return re.sub(regex, replace_string, input_data, flags=re.IGNORECASE) else: @@ -587,10 +592,11 @@ def execute_python(self, code): # 2. Subprocess execute file? try: f = StringIO() + def custom_print(*args, **kwargs): return print(*args, file=f, **kwargs) - - #with redirect_stdout(f): # just in case + + # with redirect_stdout(f): # just in case # Add globals in it too globals_copy = globals().copy() globals_copy["print"] = custom_print @@ -600,7 +606,6 @@ def custom_print(*args, **kwargs): except Exception as e: self.logger.info(f"Failed to add singul to python globals: {e}") - # Add self to globals_copy for key, value in locals().copy().items(): if key not in globals_copy: @@ -614,7 +619,7 @@ def custom_print(*args, **kwargs): # Same as a return pass except SyntaxError as e: - # Special handler for return usage. Makes return act as + # Special handler for return usage. Makes return act as # an exit() if "'return' outside function" in str(e): return { @@ -631,11 +636,11 @@ def custom_print(*args, **kwargs): # Reason: SyntaxError makes it crash BEFORE it reaches the return s = f.getvalue() - f.close() # why: https://www.youtube.com/watch?v=6SA6S9Ca5-U + f.close() # why: https://www.youtube.com/watch?v=6SA6S9Ca5-U - #try: + # try: # s = s.encode("utf-8") - #except Exception as e: + # except Exception as e: try: return { @@ -653,7 +658,7 @@ def custom_print(*args, **kwargs): "success": True, "message": s, } - + except Exception as e: return { "success": False, @@ -726,7 +731,6 @@ def preload_cache(self, key): response_data["value"] = parsed return get_response.json() - def update_cache(self, key): org_id = self.full_execution["workflow"]["execution_org"]["id"] url = f"{self.url}/api/v1/orgs/{org_id}/set_cache" @@ -743,9 +747,7 @@ def update_cache(self, key): self.cache_update_buffer = [] return get_response.json() - def filter_list(self, input_list, field, check, value, opposite): - # Remove hashtags on the fly # E.g. #.fieldname or .#.fieldname @@ -754,20 +756,22 @@ def filter_list(self, input_list, field, check, value, opposite): flip = True try: - #input_list = eval(input_list) # nosec + # input_list = eval(input_list) # nosec input_list = json.loads(input_list) except Exception: try: input_list = input_list.replace("'", '"', -1) input_list = json.loads(input_list) except Exception: - self.logger.info("[WARNING] Error parsing string to array. Continuing anyway.") + self.logger.info( + "[WARNING] Error parsing string to array. Continuing anyway." + ) # Workaround D: if not isinstance(input_list, list): return { "success": False, - "reason": "Error: input isnt a list. Please use conditions instead if using JSON.", + "reason": "Error: input isnt a list. Please use conditions instead if using JSON.", "valid": [], "invalid": [], } @@ -799,7 +803,6 @@ def filter_list(self, input_list, field, check, value, opposite): except json.decoder.JSONDecodeError as e: pass - # EQUALS JUST FOR STR if check == "equals": # Mostly for bools @@ -817,7 +820,7 @@ def filter_list(self, input_list, field, check, value, opposite): for subcheck in checklist: subcheck = str(subcheck).strip() - #ext.lower().strip() == value.lower().strip() + # ext.lower().strip() == value.lower().strip() if type(tmp) == list and subcheck in tmp: new_list.append(item) found = True @@ -874,8 +877,10 @@ def filter_list(self, input_list, field, check, value, opposite): # CONTAINS FIND FOR LIST AND IN FOR STR elif check == "contains": - #if str(value).lower() in str(tmp).lower(): - if str(value).lower() in str(tmp).lower() or self.check_wildcard(value, tmp): + # if str(value).lower() in str(tmp).lower(): + if str(value).lower() in str(tmp).lower() or self.check_wildcard( + value, tmp + ): new_list.append(item) else: failed_list.append(item) @@ -885,7 +890,9 @@ def filter_list(self, input_list, field, check, value, opposite): checklist = value.split(",") found = False for checker in checklist: - if str(checker).lower() in str(tmp).lower() or self.check_wildcard(checker, tmp): + if str(checker).lower() in str( + tmp + ).lower() or self.check_wildcard(checker, tmp): new_list.append(item) found = True break @@ -929,9 +936,9 @@ def filter_list(self, input_list, field, check, value, opposite): failed_list.append(item) elif check == "less than": # Old - #if int(tmp) < int(value): + # if int(tmp) < int(value): # new_list.append(item) - #else: + # else: # failed_list.append(item) list_set = False @@ -970,16 +977,17 @@ def filter_list(self, input_list, field, check, value, opposite): else: failed_list.append(item) - if len(self.cache_update_buffer) > 400 or (item == input_list[-1] and len(self.cache_update_buffer) > 0): + if len(self.cache_update_buffer) > 400 or ( + item == input_list[-1] and len(self.cache_update_buffer) > 0 + ): self.update_cache(value) - - #return { + # return { # "success": True, # "found": False, # "key": key, # "value": new_value, - #} + # } # SINGLE ITEM COULD BE A FILE OR A LIST OF FILES elif check == "files by extension": @@ -989,7 +997,7 @@ def filter_list(self, input_list, field, check, value, opposite): for file_id in tmp: filedata = self.get_file(file_id) _, ext = os.path.splitext(filedata["filename"]) - if (ext.lower().strip() == value.lower().strip()): + if ext.lower().strip() == value.lower().strip(): file_list.append(file_id) # else: # failed_list.append(file_id) @@ -1023,11 +1031,11 @@ def filter_list(self, input_list, field, check, value, opposite): failed_list = tmplist try: - data ={ - "success": True, - "valid": new_list, - "invalid": failed_list, - } + data = { + "success": True, + "valid": new_list, + "invalid": failed_list, + } return json.dumps(data) # new_list = json.dumps(new_list) @@ -1041,7 +1049,7 @@ def filter_list(self, input_list, field, check, value, opposite): return new_list - #def multi_list_filter(self, input_list, field, check, value): + # def multi_list_filter(self, input_list, field, check, value): # input_list = input_list.replace("'", '"', -1) # input_list = json.loads(input_list) @@ -1129,13 +1137,13 @@ def create_file(self, filename, data, category=""): try: if str(data).startswith("b'") and str(data).endswith("'"): data = data[2:-1] - if str(data).startswith("\"") and str(data).endswith("\""): + if str(data).startswith('"') and str(data).endswith('"'): data = data[2:-1] except Exception as e: self.logger.info(f"Exception: {e}") try: - #if not isinstance(data, str) and not isinstance(data, int) and not isinstance(float) and not isinstance(data, bool): + # if not isinstance(data, str) and not isinstance(data, int) and not isinstance(float) and not isinstance(data, bool): if isinstance(data, dict) or isinstance(data, list): data = json.dumps(data) except: @@ -1152,13 +1160,13 @@ def create_file(self, filename, data, category=""): if len(fileret) == 1: value = {"success": True, "filename": filename, "file_id": fileret[0]} - return value + return value - # Input is WAS a file, hence it didn't get the files + # Input is WAS a file, hence it didn't get the files def list_file_category_ids(self, file_category): return self.get_file_category_ids(file_category) - # Input is WAS a file, hence it didn't get the files + # Input is WAS a file, hence it didn't get the files # Category doesn't matter as it uses file ID, which is unique anyway def get_file_value(self, filedata, category=""): filedata = self.get_file(filedata, category) @@ -1218,7 +1226,6 @@ def download_remote_file(self, url, custom_filename="", category=""): return value - def extract_archive(self, file_id, fileformat="zip", password=None): try: return_data = {"success": False, "files": []} @@ -1227,7 +1234,6 @@ def extract_archive(self, file_id, fileformat="zip", password=None): return_ids = None with tempfile.TemporaryDirectory() as tmpdirname: - # Get archive and save phisically with open(os.path.join(tmpdirname, "archive"), "wb") as f: f.write(item["data"]) @@ -1237,7 +1243,9 @@ def extract_archive(self, file_id, fileformat="zip", password=None): # Zipfile for zipped archive if fileformat.strip().lower() == "zip": try: - with zipfile.ZipFile(os.path.join(tmpdirname, "archive")) as z_file: + with zipfile.ZipFile( + os.path.join(tmpdirname, "archive") + ) as z_file: if password: z_file.setpassword(bytes(password.encode())) @@ -1248,7 +1256,10 @@ def extract_archive(self, file_id, fileformat="zip", password=None): source = z_file.open(member) to_be_uploaded.append( - {"filename": source.name.split("/")[-1], "data": source.read()} + { + "filename": source.name.split("/")[-1], + "data": source.read(), + } ) return_data["success"] = True @@ -1276,7 +1287,10 @@ def extract_archive(self, file_id, fileformat="zip", password=None): source = z_file.open(member) to_be_uploaded.append( - {"filename": source.name.split("/")[-1], "data": source.read()} + { + "filename": source.name.split("/")[-1], + "data": source.read(), + } ) return_data["success"] = True @@ -1319,7 +1333,9 @@ def extract_archive(self, file_id, fileformat="zip", password=None): ) elif fileformat.strip().lower() == "tar.gz": try: - with tarfile.open(os.path.join(tmpdirname, "archive"), mode="r:gz") as z_file: + with tarfile.open( + os.path.join(tmpdirname, "archive"), mode="r:gz" + ) as z_file: for member in z_file.getnames(): member_files = z_file.extractfile(member) @@ -1444,22 +1460,23 @@ def create_archive(self, file_ids, fileformat, name, password=None): # Create archive temporary with tempfile.NamedTemporaryFile() as archive: - if fileformat == "zip": archive_name = "archive.zip" if not name else name - pwd = password if isinstance(password, (bytes, bytearray)) else password.encode() + pwd = ( + password + if isinstance(password, (bytes, bytearray)) + else password.encode() + ) with pyzipper.AESZipFile( - archive.name, - "w", - compression=pyzipper.ZIP_DEFLATED - ) as zf: - zf.setpassword(pwd) - zf.setencryption(pyzipper.WZ_AES, nbits=256) + archive.name, "w", compression=pyzipper.ZIP_DEFLATED + ) as zf: + zf.setpassword(pwd) + zf.setencryption(pyzipper.WZ_AES, nbits=256) - for path in paths: - zf.write(path, arcname=os.path.basename(path)) + for path in paths: + zf.write(path, arcname=os.path.basename(path)) elif fileformat == "7zip": archive_name = "archive.7z" if not name else name @@ -1491,8 +1508,13 @@ def create_archive(self, file_ids, fileformat, name, password=None): return {"success": False, "message": excp} def add_list_to_list(self, list_one, list_two): - if not isinstance(list_one, list) and not isinstance(list_one, dict): - if not list_one or list_one == " " or list_one == "None" or list_one == "null": + if not isinstance(list_one, list) and not isinstance(list_one, dict): + if ( + not list_one + or list_one == " " + or list_one == "None" + or list_one == "null" + ): list_one = "[]" try: @@ -1503,11 +1525,16 @@ def add_list_to_list(self, list_one, list_two): else: return { "success": False, - "reason": f"List one is not a valid list: {list_one}" + "reason": f"List one is not a valid list: {list_one}", } if not isinstance(list_two, list) and not isinstance(list_two, dict): - if not list_two or list_two == " " or list_two == "None" or list_two == "null": + if ( + not list_two + or list_two == " " + or list_two == "None" + or list_two == "null" + ): list_two = "[]" try: @@ -1518,7 +1545,7 @@ def add_list_to_list(self, list_one, list_two): else: return { "success": False, - "reason": f"List two is not a valid list: {list_two}" + "reason": f"List two is not a valid list: {list_two}", } if isinstance(list_one, dict): @@ -1536,19 +1563,13 @@ def diff_lists(self, list_one, list_two): try: list_one = json.loads(list_one) except json.decoder.JSONDecodeError as e: - return { - "success": False, - "reason": "list_one is not a valid list." - } + return {"success": False, "reason": "list_one is not a valid list."} if isinstance(list_two, str): try: list_two = json.loads(list_two) except json.decoder.JSONDecodeError as e: - return { - "success": False, - "reason": "list_two is not a valid list." - } + return {"success": False, "reason": "list_two is not a valid list."} def diff(li1, li2): try: @@ -1557,7 +1578,7 @@ def diff(li1, li2): # Bad json diffing - at least order doesn't matter :) not_found = [] for item in list_one: - #item = sorted(item.items()) + # item = sorted(item.items()) if item in list_two: pass else: @@ -1585,8 +1606,14 @@ def diff(li1, li2): "diff": newdiff, } - - def merge_lists(self, list_one, list_two, set_field="", sort_key_list_one="", sort_key_list_two=""): + def merge_lists( + self, + list_one, + list_two, + set_field="", + sort_key_list_one="", + sort_key_list_two="", + ): if isinstance(list_one, str): try: list_one = json.loads(list_one) @@ -1603,23 +1630,34 @@ def merge_lists(self, list_one, list_two, set_field="", sort_key_list_one="", so if isinstance(list_one, dict) and isinstance(list_two, dict): for key, value in list_two.items(): list_one[key] = value - + return list_one - return {"success": False, "message": "Both input lists need to be valid JSON lists."} + return { + "success": False, + "message": "Both input lists need to be valid JSON lists.", + } if len(list_one) != len(list_two): - return {"success": False, "message": "Lists length must be the same. %d vs %d. Are you trying to add them to a single list? Use add_list_to_list" % (len(list_one), len(list_two))} + return { + "success": False, + "message": "Lists length must be the same. %d vs %d. Are you trying to add them to a single list? Use add_list_to_list" + % (len(list_one), len(list_two)), + } if len(sort_key_list_one) > 0: try: - list_one = sorted(list_one, key=lambda k: k.get(sort_key_list_one), reverse=True) + list_one = sorted( + list_one, key=lambda k: k.get(sort_key_list_one), reverse=True + ) except: pass if len(sort_key_list_two) > 0: try: - list_two = sorted(list_two, key=lambda k: k.get(sort_key_list_two), reverse=True) + list_two = sorted( + list_two, key=lambda k: k.get(sort_key_list_two), reverse=True + ) except: pass @@ -1633,7 +1671,11 @@ def merge_lists(self, list_one, list_two, set_field="", sort_key_list_one="", so list_one[i][key] = value elif isinstance(list_two[i], str) and list_two[i] == "": continue - elif isinstance(list_two[i], str) or isinstance(list_two[i], int) or isinstance(list_two[i], bool): + elif ( + isinstance(list_two[i], str) + or isinstance(list_two[i], int) + or isinstance(list_two[i], bool) + ): if len(set_field) == 0: list_one[i][base_key] = list_two[i] else: @@ -1648,8 +1690,21 @@ def merge_lists(self, list_one, list_two, set_field="", sort_key_list_one="", so return list_one - def merge_json_objects(self, list_one, list_two, set_field="", sort_key_list_one="", sort_key_list_two=""): - return self.merge_lists(list_one, list_two, set_field=set_field, sort_key_list_one=sort_key_list_one, sort_key_list_two=sort_key_list_two) + def merge_json_objects( + self, + list_one, + list_two, + set_field="", + sort_key_list_one="", + sort_key_list_two="", + ): + return self.merge_lists( + list_one, + list_two, + set_field=set_field, + sort_key_list_one=sort_key_list_one, + sort_key_list_two=sort_key_list_two, + ) def fix_json(self, json_data): try: @@ -1670,14 +1725,14 @@ def fix_json(self, json_data): else: json_data[key] = value - #elif isinstance(value, list): + # elif isinstance(value, list): # json_data[key] = value - #else: + # else: # json_data[key] = value # #for item in json_data[key]: # # if isinstance(item, dict): # # json_data[ - + for key in deletekeys: del json_data[key] @@ -1695,7 +1750,7 @@ def xml_json_convertor(self, convertto, data): try: if convertto == "json": - data = data.replace(" encoding=\"utf-8\"", " ") + data = data.replace(' encoding="utf-8"', " ") ans = xmltodict.parse(data) ans = self.fix_json(ans) json_data = json.dumps(ans) @@ -1705,11 +1760,7 @@ def xml_json_convertor(self, convertto, data): ans = readfromstring(data) return json2xml.Json2xml(ans, wrapper="all", pretty=True).to_xml() except Exception as e: - return { - "success": False, - "input": data, - "reason": f"{e}" - } + return {"success": False, "input": data, "reason": f"{e}"} def date_to_epoch(self, input_data, date_field, date_format): if isinstance(input_data, str): @@ -1727,9 +1778,9 @@ def date_to_epoch(self, input_data, date_field, date_format): def compare_relative_date( self, timestamp, date_format, equality_test, offset, units, direction ): - if timestamp== "None": + if timestamp == "None": return False - + if date_format == "autodetect": input_dt = dateutil_parser(timestamp).replace(tzinfo=None) elif date_format != "%s": @@ -1751,7 +1802,7 @@ def compare_relative_date( if utc_format.endswith("%z"): utc_format = utc_format.replace("%z", "Z") - #if date_format != "%s" and date_format != "autodetect": + # if date_format != "%s" and date_format != "autodetect": if date_format == "autodetect": formatted_dt = datetime.datetime.utcnow() + delta elif date_format != "%s": @@ -1766,24 +1817,24 @@ def compare_relative_date( comparison_dt = formatted_dt elif direction == "ago": comparison_dt = formatted_dt - delta - #formatted_dt - delta - #comparison_dt = datetime.datetime.utcnow() + # formatted_dt - delta + # comparison_dt = datetime.datetime.utcnow() else: comparison_dt = formatted_dt + delta - #comparison_dt = datetime.datetime.utcnow() + # comparison_dt = datetime.datetime.utcnow() diff = int((input_dt - comparison_dt).total_seconds()) if units == "seconds": diff = diff elif units == "minutes": - diff = int(diff/60) + diff = int(diff / 60) elif units == "hours": - diff = int(diff/3600) + diff = int(diff / 3600) elif units == "days": - diff = int(diff/86400) + diff = int(diff / 86400) elif units == "week": - diff = int(diff/604800) + diff = int(diff / 604800) result = False if equality_test == ">": @@ -1797,7 +1848,7 @@ def compare_relative_date( result = not (result) elif equality_test == "=": - result = diff == 0 + result = diff == 0 elif equality_test == "!=": result = diff != 0 @@ -1813,7 +1864,7 @@ def compare_relative_date( parsed_string = "%s %s %s %s" % (equality_test, offset, units, direction) newdiff = diff if newdiff < 0: - newdiff = newdiff*-1 + newdiff = newdiff * -1 return { "success": True, @@ -1821,11 +1872,10 @@ def compare_relative_date( "check": parsed_string, "result": result, "diff": { - "days": int(int(newdiff)/86400), + "days": int(int(newdiff) / 86400), }, } - def run_math_operation(self, operation): result = eval(operation) return result @@ -1840,10 +1890,14 @@ def escape_html(self, input_data): result = markupsafe.escape(mapping) return mapping - def check_datastore_contains(self, key, value, append, category=""): - return check_cache_contains(self, key, value, append, category) + def check_datastore_contains( + self, key, value, append, category="", return_values="true" + ): + return check_cache_contains(self, key, value, append, category, return_values) - def check_cache_contains(self, key, value, append, category=""): + def check_cache_contains( + self, key, value, append, category="", return_values="true" + ): org_id = self.full_execution["workflow"]["execution_org"]["id"] url = "%s/api/v1/orgs/%s/get_cache" % (self.url, org_id) data = { @@ -1862,7 +1916,10 @@ def check_cache_contains(self, key, value, append, category=""): allvalues = {} try: for item in self.local_storage: - if item["execution_id"] == self.current_execution_id and item["key"] == key: + if ( + item["execution_id"] == self.current_execution_id + and item["key"] == key + ): # Max keeping the local cache properly for 5 seconds due to workflow continuations elapsed_time = time.time() - item["time_set"] if elapsed_time > 5: @@ -1871,7 +1928,10 @@ def check_cache_contains(self, key, value, append, category=""): allvalues = item["data"] except Exception as e: - print("[ERROR] Failed cache contains for current execution id local storage: %s" % e) + print( + "[ERROR] Failed cache contains for current execution id local storage: %s" + % e + ) if isinstance(value, dict) or isinstance(value, list): try: @@ -1887,15 +1947,17 @@ def check_cache_contains(self, key, value, append, category=""): if str(append).lower() == "true": append = True else: - append = False + append = False + + include_values = str(return_values).lower() == "true" if "success" not in allvalues: - #get_response = requests.post(url, json=data, verify=False) + # get_response = requests.post(url, json=data, verify=False) pass try: if "success" not in allvalues: - #allvalues = get_response.json() + # allvalues = get_response.json() allvalues = self.shared_cache if "success" not in allvalues: @@ -1925,20 +1987,21 @@ def check_cache_contains(self, key, value, append, category=""): allvalues = set_response.json() self.shared_cache = self.preload_cache(key=key) - newvalue = data["value"] try: newvalue = json.loads(data["value"]) except json.JSONDecodeError: pass - return { + response = { "success": True, "found": False, "key": key, "search": value, - "value": newvalue, } + if include_values: + response["value"] = newvalue + return response except Exception as e: return { "success": False, @@ -1976,8 +2039,8 @@ def check_cache_contains(self, key, value, append, category=""): try: for item in parsedvalue: - #return "%s %s" % (item, value) - #self.logger.info(f"{item} == {value}") + # return "%s %s" % (item, value) + # self.logger.info(f"{item} == {value}") if str(item) == str(value): if not append: try: @@ -1986,25 +2049,31 @@ def check_cache_contains(self, key, value, append, category=""): newdata["data"] = allvalues self.local_storage.append(newdata) except Exception as e: - print("[ERROR] Failed in local storage append: %s" % e) + print( + "[ERROR] Failed in local storage append: %s" % e + ) - return { + response = { "success": True, "found": True, "reason": "Found and not appending!", "key": key, "search": value, - "value": allvalues["value"], } + if include_values: + response["value"] = allvalues["value"] + return response else: - return { + response = { "success": True, "found": True, "reason": "Found, was appending, but item already exists", "key": key, "search": value, - "value": allvalues["value"], } + if include_values: + response["value"] = allvalues["value"] + return response # Lol break @@ -2013,20 +2082,24 @@ def check_cache_contains(self, key, value, append, category=""): append = True if not append: - return { + response = { "success": True, "found": False, "reason": "Not found, not appending (2)!", "key": key, "search": value, - "value": allvalues["value"], } + if include_values: + response["value"] = allvalues["value"] + return response - #parsedvalue.append(value) + # parsedvalue.append(value) - #data["value"] = json.dumps(parsedvalue) + # data["value"] = json.dumps(parsedvalue) - if value not in allvalues["value"] and isinstance(allvalues["value"], list): + if value not in allvalues["value"] and isinstance( + allvalues["value"], list + ): self.cache_update_buffer.append(value) allvalues["value"].append(value) @@ -2052,14 +2125,16 @@ def check_cache_contains(self, key, value, append, category=""): except: pass - return { + response = { "success": True, "found": False, - "reason": f"Appended as it didn't exist", + "reason": "Appended as it didn't exist", "key": key, "search": value, - "value": newvalue, } + if include_values: + response["value"] = newvalue + return response except Exception as e: exception = e pass @@ -2069,10 +2144,10 @@ def check_cache_contains(self, key, value, append, category=""): "found": True, "reason": f"Failed to set append the value: {exception}. This should never happen", "search": value, - "key": key + "key": key, } - #return allvalues + # return allvalues except Exception as e: print("[ERROR] Failed check cache contains: %s" % e) @@ -2084,9 +2159,8 @@ def check_cache_contains(self, key, value, append, category=""): "found": False, } - return value.text + return value.text - ## Adds value to a subkey of the cache ## subkey = "hi", value = "test", overwrite=False ## {"subkey": "hi", "value": "test"} @@ -2124,13 +2198,17 @@ def change_cache_subkey(self, key, subkey, value, overwrite, category=""): try: allvalues = response.json() allvalues["key"] = key - #allvalues["value"] = json.loads(json.dumps(value)) + # allvalues["value"] = json.loads(json.dumps(value)) - if (value.startswith("{") and value.endswith("}")) or (value.startswith("[") and value.endswith("]")): + if (value.startswith("{") and value.endswith("}")) or ( + value.startswith("[") and value.endswith("]") + ): try: allvalues["value"] = json.loads(value) except json.decoder.JSONDecodeError as e: - self.logger.info("[WARNING] Failed inner value cache parsing: %s" % e) + self.logger.info( + "[WARNING] Failed inner value cache parsing: %s" % e + ) allvalues["value"] = str(value) else: allvalues["value"] = str(value) @@ -2151,7 +2229,18 @@ def get_datastore_value(self, key, category=""): def get_ioc(self, ioc, data_type=""): if len(data_type) == 0: - ioc_types = ["domains", "urls", "email_addresses", "ipv4s", "ipv6s", "ipv4_cidrs", "md5s", "sha256s", "sha1s", "cves"] + ioc_types = [ + "domains", + "urls", + "email_addresses", + "ipv4s", + "ipv6s", + "ipv4_cidrs", + "md5s", + "sha256s", + "sha1s", + "cves", + ] iocs = find_iocs(str(ioc)) for key, value in iocs.items(): @@ -2183,8 +2272,8 @@ def get_ioc(self, ioc, data_type=""): if allvalues["success"] == True and len(allvalues["value"]) > 0: allvalues["found"] = True else: - allvalues["success"] = True - allvalues["found"] = False + allvalues["success"] = True + allvalues["found"] = False try: parsedvalue = json.loads(allvalues["value"]) @@ -2220,8 +2309,8 @@ def get_cache_value(self, key, category=""): if allvalues["success"] == True and len(allvalues["value"]) > 0: allvalues["found"] = True else: - allvalues["success"] = True - allvalues["found"] = False + allvalues["success"] = True + allvalues["found"] = False try: parsedvalue = json.loads(allvalues["value"]) @@ -2260,9 +2349,11 @@ def search_datastore_category(self, input_list, key, category): try: input_list = json.loads(str(input_list)) except Exception as e: - returnvalue["reason"] = f"Input list is not a valid JSON list: {input_list}", + returnvalue["reason"] = ( + f"Input list is not a valid JSON list: {input_list}", + ) returnvalue["details"] = str(e) - return returnvalue + return returnvalue org_id = self.full_execution["workflow"]["execution_org"]["id"] cnt = -1 @@ -2272,23 +2363,31 @@ def search_datastore_category(self, input_list, key, category): try: item = json.loads(str(item)) except Exception as e: - self.logger.info("[ERROR][%s] Failed to parse item as JSON: %s" % (self.current_execution_id, e)) + self.logger.info( + "[ERROR][%s] Failed to parse item as JSON: %s" + % (self.current_execution_id, e) + ) continue input_list[cnt] = item if key not in item: - returnvalue["reason"] = "Couldn't find key '%s' in every item. Make sure to use a key that exists in every entry." % (key), + returnvalue["reason"] = ( + "Couldn't find key '%s' in every item. Make sure to use a key that exists in every entry." + % (key), + ) return returnvalue - data.append({ - "workflow_id": self.full_execution["workflow"]["id"], - "execution_id": self.current_execution_id, - "authorization": self.authorization, - "org_id": org_id, - "key": str(item[key]), - "value": json.dumps(item), - "category": category, - }) + data.append( + { + "workflow_id": self.full_execution["workflow"]["id"], + "execution_id": self.current_execution_id, + "authorization": self.authorization, + "org_id": org_id, + "key": str(item[key]), + "value": json.dumps(item), + "category": category, + } + ) url = f"{self.url}/api/v2/datastore?bulk=true&execution_id={self.current_execution_id}&authorization={self.authorization}" response = requests.post(url, json=data, verify=False) @@ -2300,16 +2399,18 @@ def search_datastore_category(self, input_list, key, category): return returnvalue data = "" - try: + try: data = response.json() except json.decoder.JSONDecodeError as e: - return response.text + return response.text if "keys_existed" not in data: - returnvalue["error"] = "Invalid response from backend during bulk update of keys" + returnvalue["error"] = ( + "Invalid response from backend during bulk update of keys" + ) returnvalue["details"] = data - return returnvalue + return returnvalue not_found_keys = [] returnvalue["success"] = True @@ -2324,17 +2425,22 @@ def search_datastore_category(self, input_list, key, category): else: returnvalue["new"].append(datastore_item) - found = True + found = True break if not found: - print("[ERROR][%s] Key %s not found in datastore response, adding as new" % (self.current_execution_id, datastore_item[key])) - #returnvalue["new"].append(datastore_item) + print( + "[ERROR][%s] Key %s not found in datastore response, adding as new" + % (self.current_execution_id, datastore_item[key]) + ) + # returnvalue["new"].append(datastore_item) not_found_keys.append(datastore_item[key]) if len(not_found_keys) > 0: returnvalue["unhandled_keys"] = not_found_keys - returnvalue["reason"] = "Something went wrong updating the unhandled_keys. Please contact support@shuffler.io if this persists." + returnvalue["reason"] = ( + "Something went wrong updating the unhandled_keys. Please contact support@shuffler.io if this persists." + ) return json.dumps(returnvalue, indent=4) @@ -2347,7 +2453,7 @@ def set_cache_value(self, key, value, category=""): value = json.dumps(value) except Exception as e: self.logger.info(f"[WARNING] Error in JSON dumping (set cache): {e}") - + if not isinstance(value, str): value = str(value) @@ -2367,20 +2473,24 @@ def set_cache_value(self, key, value, category=""): try: allvalues = response.json() allvalues["key"] = key - #allvalues["value"] = json.loads(json.dumps(value)) + # allvalues["value"] = json.loads(json.dumps(value)) - allvalues["existed"] = False + allvalues["existed"] = False if "keys_existed" in allvalues: for key_info in allvalues["keys_existed"]: if key_info["key"] == key: allvalues["existed"] = key_info["existed"] break - if (value.startswith("{") and value.endswith("}")) or (value.startswith("[") and value.endswith("]")): + if (value.startswith("{") and value.endswith("}")) or ( + value.startswith("[") and value.endswith("]") + ): try: allvalues["value"] = json.loads(value) except json.decoder.JSONDecodeError as e: - self.logger.info("[WARNING] Failed inner value cache parsing: %s" % e) + self.logger.info( + "[WARNING] Failed inner value cache parsing: %s" % e + ) allvalues["value"] = str(value) else: allvalues["value"] = str(value) @@ -2393,12 +2503,17 @@ def set_cache_value(self, key, value, category=""): self.logger.info("Value couldn't be parsed") return response.text - def convert_json_to_tags(self, json_object, split_value=", ", include_key=True, lowercase=True): + def convert_json_to_tags( + self, json_object, split_value=", ", include_key=True, lowercase=True + ): if isinstance(json_object, str): try: json_object = json.loads(json_object) except json.decoder.JSONDecodeError as e: - self.logger.info("Failed to parse list2 as json: %s. Type: %s" % (e, type(json_object))) + self.logger.info( + "Failed to parse list2 as json: %s. Type: %s" + % (e, type(json_object)) + ) if isinstance(lowercase, str) and lowercase.lower() == "true": lowercase = True @@ -2413,7 +2528,11 @@ def convert_json_to_tags(self, json_object, split_value=", ", include_key=True, parsedstring = [] try: for key, value in json_object.items(): - if isinstance(value, str) or isinstance(value, int) or isinstance(value, bool): + if ( + isinstance(value, str) + or isinstance(value, int) + or isinstance(value, bool) + ): if include_key == True: parsedstring.append("%s:%s" % (key, value)) else: @@ -2462,7 +2581,7 @@ def cidr_ip_match(self, ip, networks): try: ip_networks = list(map(ipaddress.ip_network, networks)) - #ip_address = ipaddress.ip_address(ip, False) + # ip_address = ipaddress.ip_address(ip, False) ip_address = ipaddress.ip_address(ip) except ValueError as e: return "IP or some networks are not in valid format.\nError: {}".format(e) @@ -2471,8 +2590,8 @@ def cidr_ip_match(self, ip, networks): result = {} result["ip"] = ip - result['networks'] = list(map(str, matched_networks)) - result['is_contained'] = True if len(result['networks']) > 0 else False + result["networks"] = list(map(str, matched_networks)) + result["is_contained"] = True if len(result["networks"]) > 0 else False return json.dumps(result) @@ -2488,12 +2607,12 @@ def get_hash_sum(self, value): sha256_value = "" try: - md5_value = hashlib.md5(str(value).encode('utf-8')).hexdigest() + md5_value = hashlib.md5(str(value).encode("utf-8")).hexdigest() except Exception as e: pass try: - sha256_value = hashlib.sha256(str(value).encode('utf-8')).hexdigest() + sha256_value = hashlib.sha256(str(value).encode("utf-8")).hexdigest() except Exception as e: pass @@ -2504,14 +2623,17 @@ def get_hash_sum(self, value): "sha256": sha256_value, } - return parsedvalue + return parsedvalue def run_oauth_request(self, url, jwt): headers = { "Content-Type": "application/x-www-form-urlencoded", } - data = "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=%s" % jwt + data = ( + "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=%s" + % jwt + ) return requests.post(url, data=data, headers=headers, verify=False).text @@ -2519,10 +2641,9 @@ def run_oauth_request(self, url, jwt): def get_jwt_from_file(self, file_id, jwt_audience, scopes, complete_request=True): allscopes = scopes - if "," in scopes: allscopes = " ".join(scopes.split(",")) - + # Service account key path filedata = self.get_file(file_id) if filedata["success"] == False: @@ -2530,49 +2651,43 @@ def get_jwt_from_file(self, file_id, jwt_audience, scopes, complete_request=True "success": False, "message": f"Failed to get file for ID {file_id}", } - + data = json.loads(filedata["data"], strict=False) - #sa_keyfile = "" + # sa_keyfile = "" sa_keyfile = data["private_key"] sa_email = data["client_email"] - + # The audience to target audience = jwt_audience - + """Generates a signed JSON Web Token using a Google API Service Account or similar.""" - def get_jwt(sa_keyfile, - sa_email, - audience, - allscopes, - expiry_length=3600): - + + def get_jwt(sa_keyfile, sa_email, audience, allscopes, expiry_length=3600): now = int(time.time()) - + # build payload payload = { # expires after 'expiry_length' seconds. # iss must match 'issuer' in the security configuration in your # swagger spec (e.g. service account email). It can be any string. - 'iss': sa_email, + "iss": sa_email, # aud must be either your Endpoints service name, or match the value # specified as the 'x-google-audience' in the OpenAPI document. - 'scope': allscopes, - 'aud': audience, + "scope": allscopes, + "aud": audience, "exp": now + expiry_length, - 'iat': now, - + "iat": now, # sub and email should match the service account's email address - 'sub': sa_email, - 'email': sa_email, + "sub": sa_email, + "email": sa_email, } - + # sign with keyfile - #signer = crypt.RSASigner.from_service_account_file(sa_keyfile) + # signer = crypt.RSASigner.from_service_account_file(sa_keyfile) signer = crypt.RSASigner.from_string(sa_keyfile) jwt_token = jwt.encode(signer, payload) return jwt_token - - + signed_jwt = get_jwt(sa_keyfile, sa_email, audience, allscopes) if str(complete_request).lower() == "true": @@ -2601,11 +2716,18 @@ def get_synonyms(self, input_type): "uuid", "teamid", "messageid", - ], - "title": ["title", "message", "subject", "name"], - "description": ["description", "status", "explanation", "story", "details", "snippet"], - "email": ["mail", "email", "sender", "receiver", "recipient"], - "data": [ + ], + "title": ["title", "message", "subject", "name"], + "description": [ + "description", + "status", + "explanation", + "story", + "details", + "snippet", + ], + "email": ["mail", "email", "sender", "receiver", "recipient"], + "data": [ "data", "ip", "domain", @@ -2617,9 +2739,9 @@ def get_synonyms(self, input_type): "value", "item", "rules", - ], - "tags": ["tags", "taxonomies", "labels", "labelids"], - "assignment": [ + ], + "tags": ["tags", "taxonomies", "labels", "labelids"], + "assignment": [ "assignment", "user", "assigned_to", @@ -2627,40 +2749,44 @@ def get_synonyms(self, input_type): "closed_by", "closing_user", "opened_by", - ], - "severity": [ + ], + "severity": [ "severity", "sev", "magnitude", "relevance", - ] + ], } - + return [] - + def find_key(self, inputkey, synonyms): inputkey = inputkey.lower().replace(" ", "").replace(".", "") for key, value in synonyms.items(): if inputkey in value: return key - + return inputkey - + def run_key_recursion(self, json_input, synonyms): - if isinstance(json_input, str) or isinstance(json_input, int) or isinstance(json_input, float): + if ( + isinstance(json_input, str) + or isinstance(json_input, int) + or isinstance(json_input, float) + ): return json_input, {} - + if isinstance(json_input, list): if len(json_input) != 1: return json_input, {} else: json_input = json_input[0] - - #new_list = [] - #for item in json_input: - #run_key_recursion(item, synonyms) - #new_dict[new_key], found_important = run_key_recursion(value, synonyms) - + + # new_list = [] + # for item in json_input: + # run_key_recursion(item, synonyms) + # new_dict[new_key], found_important = run_key_recursion(value, synonyms) + # Looks for exact key:value stuff in other format if len(json_input.keys()) == 2: newkey = "" @@ -2670,54 +2796,58 @@ def run_key_recursion(self, json_input, synonyms): newkey = value elif key == "value": newvalue = value - + if len(newkey) > 0 and len(newvalue) > 0: json_input[newkey] = newvalue try: del json_input["name"] except: pass - + try: del json_input["value"] except: pass - + try: del json_input["key"] except: pass - + important_fields = {} new_dict = {} for key, value in json_input.items(): new_key = self.find_key(key, synonyms) - + if isinstance(value, list): new_list = [] for subitem in value: - returndata, found_important = self.run_key_recursion(subitem, synonyms) - + returndata, found_important = self.run_key_recursion( + subitem, synonyms + ) + new_list.append(returndata) for subkey, subvalue in found_important.items(): - important_fields[subkey] = subvalue - + important_fields[subkey] = subvalue + new_dict[new_key] = new_list - + elif isinstance(value, dict): # FIXMe: Try to understand Key:Values as well by translating them # name/key: subject # value: This is a subject # will become: # subject: This is a subject - - new_dict[new_key], found_important = self.run_key_recursion(value, synonyms) - + + new_dict[new_key], found_important = self.run_key_recursion( + value, synonyms + ) + for subkey, subvalue in found_important.items(): important_fields[subkey] = subvalue else: new_dict[new_key] = value - + # Translated fields are added as important if key.lower().replace(" ", "").replace(".", "") != new_key: try: @@ -2727,31 +2857,33 @@ def run_key_recursion(self, json_input, synonyms): important_fields[new_key] = new_dict[new_key] except: important_fields[new_key] = new_dict[new_key] - - #break - + + # break + return new_dict, important_fields - + # Should translate the data to something more useful def get_standardized_data(self, json_input, input_type): if isinstance(json_input, str): json_input = json.loads(json_input, strict=False) - + input_synonyms = self.get_synonyms(input_type) - parsed_data, important_fields = self.run_key_recursion(json_input, input_synonyms) - + parsed_data, important_fields = self.run_key_recursion( + json_input, input_synonyms + ) + # Try base64 decoding and such too? for key, value in important_fields.items(): try: important_fields[key] = important_fields[key][key] except: pass - + try: important_fields[key] = base64.b64decode(important_fields[key]) except: pass - + return { "success": True, "original": json_input, @@ -2773,14 +2905,16 @@ def generate_random_string(self, length=16, special_characters=True): if str(special_characters).lower() == "false": characters = string.ascii_letters + string.digits + string.punctuation - password = ''.join(random.choice(characters) for i in range(length)) + password = "".join(random.choice(characters) for i in range(length)) return { "success": True, "password": password, } - - def run_ssh_command(self, host, port, user_name, private_key_file_id, password, command): + + def run_ssh_command( + self, host, port, user_name, private_key_file_id, password, command + ): ssh_client = paramiko.SSHClient() ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) @@ -2795,60 +2929,74 @@ def run_ssh_command(self, host, port, user_name, private_key_file_id, password, try: key_data = new_file["data"].decode() except Exception as e: - return {"success":"false","message":str(e)} + return {"success": "false", "message": str(e)} private_key_file = StringIO() private_key_file.write(key_data) private_key_file.seek(0) private_key = paramiko.RSAKey.from_private_key(private_key_file) - + try: - ssh_client.connect(hostname=host,username=user_name,port=port, pkey= private_key) + ssh_client.connect( + hostname=host, username=user_name, port=port, pkey=private_key + ) except Exception as e: - return {"success":"false","message":str(e)} + return {"success": "false", "message": str(e)} else: try: - ssh_client.connect(hostname=host,username=user_name,port=port, password=str(password)) + ssh_client.connect( + hostname=host, username=user_name, port=port, password=str(password) + ) except Exception as e: - return {"success":"false","message":str(e)} + return {"success": "false", "message": str(e)} try: stdin, stdout, stderr = ssh_client.exec_command(str(command)) try: - errorLog = stderr.read().decode(errors='ignore') + errorLog = stderr.read().decode(errors="ignore") except Exception as e: errorLog = f"Failed to read stderr {e}" try: - output = stdout.read().decode(errors='ignore') + output = stdout.read().decode(errors="ignore") except Exception as e: output = f"Failed to read stdout {e}" except Exception as e: - return {"success":"false","message":str(e)} + return {"success": "false", "message": str(e)} - return {"success":"true","output": output, "error_logs": errorLog} + return {"success": "true", "output": output, "error_logs": errorLog} def cleanup_ioc_data(self, input_data): # Remove unecessary parts like { and }, quotes etc input_data = str(input_data) input_data = input_data.replace("{", "") input_data = input_data.replace("}", "") - input_data = input_data.replace("\"", "") + input_data = input_data.replace('"', "") input_data = input_data.replace("'", "") input_data = input_data.replace("\t", " ") input_data = input_data.replace(" ", " ") input_data = input_data.replace("\n\n", "\n") - # Remove html tags - input_data = re.sub(r'<[^>]*>', '', input_data) + # Remove html tags + input_data = re.sub(r"<[^>]*>", "", input_data) return input_data - def parse_ioc(self, input_string, input_type="all"): - ioc_types = ["domains", "urls", "email_addresses", "ipv4s", "ipv6s", "ipv4_cidrs", "md5s", "sha256s", "sha1s", "cves"] - #ioc_types = ["ipv4s"] + ioc_types = [ + "domains", + "urls", + "email_addresses", + "ipv4s", + "ipv6s", + "ipv4_cidrs", + "md5s", + "sha256s", + "sha1s", + "cves", + ] + # ioc_types = ["ipv4s"] try: input_string = self.cleanup_ioc_data(input_string) @@ -2874,21 +3022,24 @@ def parse_ioc(self, input_string, input_type="all"): new_input_types.append(item) - ioc_types = new_input_types + ioc_types = new_input_types if len(ioc_types) == 0: input_type = "all" # Not used for anything after cleanup fixes - max_size = 7500000 - #if len(input_string) > max_size: + max_size = 7500000 + # if len(input_string) > max_size: # input_string = input_string[:max_size] - self.logger.info("[DEBUG] Parsing data of length %d with types %s. Max size: %d" % (len(input_string), ioc_types, max_size)) + self.logger.info( + "[DEBUG] Parsing data of length %d with types %s. Max size: %d" + % (len(input_string), ioc_types, max_size) + ) self.logger.info(f"STRING: {input_string}") - #iocs = find_iocs(str(input_string), included_ioc_types=ioc_types) + # iocs = find_iocs(str(input_string), included_ioc_types=ioc_types) iocs = find_iocs(str(input_string)) - #self.logger.info("[DEBUG] Found %d ioc types" % len(iocs)) + # self.logger.info("[DEBUG] Found %d ioc types" % len(iocs)) newarray = [] for key, value in iocs.items(): @@ -2925,7 +3076,9 @@ def parse_ioc(self, input_string, input_type="all"): elif "ip" in item["data_type"]: item["data_type"] = "ip" try: - item["is_private_ip"] = ipaddress.ip_address(item["data"]).is_private + item["is_private_ip"] = ipaddress.ip_address( + item["data"] + ).is_private except: pass @@ -2935,25 +3088,24 @@ def parse_ioc(self, input_string, input_type="all"): return "Failed to parse IOC's: %s" % e return newarray - def split_text(self, text): # Split text into chunks of 10kb. Add each 10k to array # In case e.g. 1.2.3.4 lands exactly on 20k boundary, it may be useful to overlap here. # (just shitty code to reduce chance of issues) while still going fast arr_one = [] - max_len = 5000 + max_len = 5000 current_string = "" - overlaps = 100 + overlaps = 100 for i in range(0, len(text)): current_string += text[i] if len(current_string) > max_len: # Appending just in case even with overlaps - if len(text) > i+overlaps: - current_string += text[i+1:i+overlaps] + if len(text) > i + overlaps: + current_string += text[i + 1 : i + overlaps] else: - current_string += text[i+1:] + current_string += text[i + 1 :] arr_one.append(current_string) current_string = "" @@ -2961,13 +3113,13 @@ def split_text(self, text): if len(current_string) > 0: arr_one.append(current_string) - return arr_one + return arr_one def _format_result(self, result): final_result = {} - + for res in result: - for key,val in res.items(): + for key, val in res.items(): if key in final_result: if isinstance(val, list) and len(val) > 0: for i in val: @@ -2975,7 +3127,7 @@ def _format_result(self, result): elif isinstance(val, dict): if key in final_result: if isinstance(val, dict): - for k,v in val.items(): + for k, v in val.items(): val[k].append(v) else: final_result[key] = val @@ -2985,36 +3137,49 @@ def _format_result(self, result): # See function for how it works~: parse_ioc_new(..) def _with_concurency(self, array_of_strings, ioc_types): results = [] - #start = time.perf_counter() + # start = time.perf_counter() # Workers dont matter..? - # What can we use instead? + # What can we use instead? workers = 4 with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor: # Submit the find_iocs function for each string in the array - futures = [executor.submit( - find_iocs, - text=string, - included_ioc_types=ioc_types, - ) for string in array_of_strings] + futures = [ + executor.submit( + find_iocs, + text=string, + included_ioc_types=ioc_types, + ) + for string in array_of_strings + ] # Wait for all tasks to complete concurrent.futures.wait(futures) # Retrieve the results if needed results = [future.result() for future in futures] - + return self._format_result(results) # FIXME: Make this good and actually faster than normal # For now: Concurrency doesn't make it faster due to GIL in python. - # May need to offload this to an executable or something + # May need to offload this to an executable or something def parse_ioc_new(self, input_string, input_type="all"): if input_type == "": input_type = "all" - ioc_types = ["domains", "urls", "email_addresses", "ipv4s", "ipv4_cidrs", "md5s", "sha256s", "sha1s", "cves"] + ioc_types = [ + "domains", + "urls", + "email_addresses", + "ipv4s", + "ipv4_cidrs", + "md5s", + "sha256s", + "sha1s", + "cves", + ] if input_type == "" or input_type == "all": ioc_types = ioc_types @@ -3028,7 +3193,9 @@ def parse_ioc_new(self, input_string, input_type="all"): input_string = str(input_string) if len(input_string) > 10000: - iocs = self._with_concurency(self.split_text(input_string), ioc_types=ioc_types) + iocs = self._with_concurency( + self.split_text(input_string), ioc_types=ioc_types + ) else: iocs = find_iocs(input_string, included_ioc_types=ioc_types) @@ -3037,7 +3204,7 @@ def parse_ioc_new(self, input_string, input_type="all"): if input_type != "all": if key not in input_type: continue - + if len(value) == 0: continue @@ -3071,7 +3238,9 @@ def parse_ioc_new(self, input_string, input_type="all"): newarray[i]["data_type"] = "ip" try: - newarray[i]["is_private_ip"] = ipaddress.ip_address(item["data"]).is_private + newarray[i]["is_private_ip"] = ipaddress.ip_address( + item["data"] + ).is_private except Exception as e: pass @@ -3085,15 +3254,12 @@ def parse_ioc_new(self, input_string, input_type="all"): def merge_incoming_branches(self, input_type="list"): wf = self.full_execution["workflow"] if "branches" not in wf or not wf["branches"]: - return { - "success": False, - "reason": "No branches found" - } + return {"success": False, "reason": "No branches found"} if "results" not in self.full_execution or not self.full_execution["results"]: return { "success": False, - "reason": "No results for previous actions not found" + "reason": "No results for previous actions not found", } if not input_type: @@ -3101,7 +3267,7 @@ def merge_incoming_branches(self, input_type="list"): branches = wf["branches"] cur_action = self.action - #print("Found %d branches" % len(branches)) + # print("Found %d branches" % len(branches)) results = [] for branch in branches: @@ -3134,13 +3300,13 @@ def merge_incoming_branches(self, input_type="list"): continue newlist.append(subitem) - #newlist.append(item) + # newlist.append(item) results = newlist elif input_type == "dict": new_dict = {} for item in results: - if not isinstance(item, dict): + if not isinstance(item, dict): continue new_dict = self.merge_lists(new_dict, item) @@ -3149,7 +3315,7 @@ def merge_incoming_branches(self, input_type="list"): else: return { "success": False, - "reason": "No results from source branches with type %s" % input_type + "reason": "No results from source branches with type %s" % input_type, } return results @@ -3158,10 +3324,7 @@ def bodyparse_test(self, body): return body def list_cidr_ips(self, cidr): - defaultreturn = { - "success": False, - "reason": "Invalid CIDR address" - } + defaultreturn = {"success": False, "reason": "Invalid CIDR address"} if not cidr: return defaultreturn @@ -3187,11 +3350,7 @@ def list_cidr_ips(self, cidr): return defaultreturn ips = [str(ip) for ip in net] - returnvalue = { - "success": True, - "amount": len(ips), - "ips": ips - } + returnvalue = {"success": True, "amount": len(ips), "ips": ips} return returnvalue @@ -3214,12 +3373,13 @@ def switch(self, conditions): # Loop conditions # Return them without a loop to make it EASY to understand - # Validation should be: + # Validation should be: # Continuation based on .id.valid # .valid -> true/false # If no id exists, use name? return to_return + if __name__ == "__main__": Tools.run()