Shuffle extensions
This is documentation for integrating and sending data from third-party services to Shuffle. Not to be confused with apps and workflows
Table of contents
Introduction
From the start, Shuffle has been a platform about integrations. We've focused on making them as open and usable as possible, but were missing one part; inbound data. The general way Shuffle handles this has been through third-party API's, where we poll for data on a schedule. There are however some cases where this doesn't do the trick. That's what extensions are.
These integrations will typically entail third party services connecting to Shuffle with inbound Webhooks as triggers in a workflow.
Single Signon
Shuffle added Single Signon (SAML v1.0) from Shuffle version 0.9.16 & OpenID in Shuffle version 1.0.0. This allows you to log into Shuffle from other identity platforms, entirely controlled by you. SSO is available for onprem AND cloud. It works by setting an Entrypoint (IdP) and X509 Certificate, both used to validate who the requester is. This can be added in your admin panel.
Org Swapping behavior
If an Organization requires SSO, it will FORCE you through the SSO login unless your session already has been through SSO for that organization. This may feel counter-intuitive at first, but is a required system as each organization is controlled for SSO individually, and there is no limit to how many organizations a user can have.
Using ANY SSO platform
ONPREM ONLY: You will have to change the SSO_REDIRECT_URL variable in the .env file to match your front end server link i.e SSO_REDIRECT_URL=http://<URL>:<PORT>
How it works:
- Every user gets the role "user" by default.
- If the user DOESN'T exist, it will be created, based on an identifier provided from the SSO provider.
- If the user DOES exist, it will simply log them in.
- After login, each user is redirected to /workflows as per a normal login.
PS: In some cases, the persons' username may be appear an ID in Shuffle. If so, an admin should login to Shuffle and change their username. SSO should still work.
PPS: The callback URL/redirect URL/recipient URL is https://<URL>:<PORT>/api/v1/login_sso
Okta
To use Okta SSO with Shuffle, first make an app in Okta. Their guide for making an app can be found here.
Once an application is made, it's time to find the required information. Go to the app > Sign On > Click "View Setup Instructions" under SAML 2.0. This will open a new page with the credentials.
Move these fields over to Shuffle:
- Identity Provider Single Sign-On URL -> SSO Entrypoint (IdP)
- X.509 -> SSO Certificate (X509)
After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization.
Google SAML SSO
• Log in to your Google Workspace portal as admin -> Apps -> Web and mobile apps.
• Click Add app -> Add custom SAML app.
• In the popup window, enter shuffle as the name of the app, you can upload the app icon here as well. Then click continue.
• The next page will provide the IDP data. Download IdP metadata then click continue.
• Next page will be the SP information, this is where you should provide the Single Sign On URL, and SP Entity ID.
- ACS URL : - For Cloud : https://shuffler.io/api/v1/login_sso
- For On-prem: https://
• The next page is where you add the below attributes and click on Finish and the app is added to the Google Workspace.
• After completing the app creation on Google Workspace. You can select the shuffle app under the "Web and mobile apps". Click on the shuffle app, and note the SPID in the URL, that is the SPID that will be needed.
• Now we configure the Google SAML SSO in shuffle portal. Log in to your Shuffle web portal.
• Click on profile -> Admin --> Edit Details --> Now add the Entrypoint (IdP) and SSO Certificate under the SAML SSO and click on save.
Note : The IdP metadata contains the Idp ID and certificate. Use this link as your SSO entry point, substituting your SP ID for "YourSPID" and your IDP ID for "YourIDPId."
- SSO Entrypoint (IdP) : https://accounts.google.com/o/saml2/initsso?idpid=C0ivw&spid=634181550209&forceauthn=falseAuth0
To use Auth0 SSO with Shuffle, first make an app on https://manage.auth0.com. Documentation can be found here.
After the app is made, click "Addons" > "SAML2 Web App".
In the popup, move the data of these fields to Shuffle:
- Identity Provider Login URL -> SSO Entrypoint (IdP)
- Identity Provider Certificate -> SSO Certificate (X509)
Open the Certificate file in a text editor, and copy it's contents.
After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization.
PingIdentity
To use PingID SSO with Shuffle, first make an app on https://console.pingone.eu/. Documentation can be found here.
After the app is made, click the dropdown for it on the right side > Configuration > find these fields.
In the view above, move the data of these fields to Shuffle:
- INITIATE SINGLE SIGN-ON URL -> SSO Entrypoint (IdP)
- DOWNLOAD METADATA -> SSO Certificate (X509)
Open the Certificate file in a text editor, and copy it's contents in the field.
After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization.
Keycloak
Open ID SSO setup with Keycloak:
- Click on clients
- Create new client
- Set client id
- Ensure client protocol is openid-connect
- Hit save
You now have a client. Click on settings and configure them as follows -Your client ID -Set up a name -Client Protocol set to openid-connect -Access Type set to public -Standard Flow Enabled toggled to ON -Direct access grants toggled to ON
For the Valid Redirect URI http://
Under the Fine Grain OpenID Connect Configuration
Valid Request URIs http://
Once this is done head over to your shuffle instance.
1. Click on Admin button
2. Scroll down and click on the downward facing arrow beneath Organization overview
3. Scroll down again to OpenID connect
4. Fill in the client ID (It should be the same as what you entered in Keycloak)
5. Authorization URL http://<Your_Keycloak_URL>:
If you keep getting redirected to your backend url, head on to your Shuffle folder on your server. 1. Vim .env
- In vim change the BASE URL to your server link; i.e SSO_REDIRECT_URL=http://
:
Finally go back to shuffle and use SSO button to login.
Azure AD
To use OpenID with Azure AD, Shuffle supports OpenID connect with the use of Client IDs and Client secrets. To set up OpenID Connect with Azure, we use "ID_token" authentication. This entails a few normal steps regarding app creation in Azure App Registration.
Set up an app in Azure AD with ID tokens enabled Go to app registrations and create a new app. Make it use "Web" for redirect URI's and direct it to your Shuffle instance at /api/v1/login_openid. From here, make sure to go to "authentication" and enable "ID Tokens"
Get the Client ID, Client Secret and your Tenant Start by generating a client secret. Keep it safe, as we'll use it later.
The client secret and tenant ID can be found in the "Overview" tab:
Go to the admin panel in Shuffle and put in the Client ID and Secret
Put in your Tenant ID in the authorization URL The URL is as such:
https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/authorize. The Token URL is not strictly required for ID Token auth.
In the "Token Configuration" tab on the left, click "Add Optional Claim." Select the token type as "ID" and choose "email" as the claim, then click "Add."
Done! Click save and log out. Try your new login based on your Azure AD configuration.
PS: When a user signs in, they are granted the access rights of a "user" within the designated organization. The username will be derived from the email address listed in the Azure users list. Therefore, when creating or adding a new user in Azure, ensure that the email field in their profile is populated. If this email field is empty, the user will not be able to log in to the organization.
Other Platforms
As long as you can create an identity and acquire an Entrypoint (IdP) and X509, paste them into the Shuffle fields, and it should work with any SAML/SSO provider.
Testing SSO
Testing Single Sign-On (SSO) ensures that your SSO setup is functioning correctly. When you click the "Test SSO" button, you will be redirected to your SSO authentication page. After authentication, you will be redirected to https://shuffler.io/workflows if you are using the cloud version, or https://frontend-url/workflows if you are using the on-premises version. There are two test cases to verify the SSO feature:
- User Change During Authentication: If you log into your Shuffle account with username "A" and authenticate with username "B", your user will change to "B". If user "B" does not exist in the organization, they will be added. The video below demonstrates logging into Shuffle with "lalitdeore12@gmail.com" and authenticating with Azure username "lalitdeoretest@lalitdeore12gmail.onmicrosoft.com" (which has the email "lalit@shuffler.io"). After authentication, the user changes from "lalitdeore12@gmail.com" to "lalit@shuffler.io".
https://github.com/user-attachments/assets/8c3474a5-bfdd-4c68-bd59-0b7b1ddb2b0c
- User Remains the Same During Authentication: If you log into Shuffle with user "A" and authenticate with the same user "A", your user will not change. The video below shows logging into Shuffle with "lalitdeore12@gmail.com" and authenticating with the same user in Azure. After authentication, the user remains unchanged.
https://github.com/user-attachments/assets/0a927283-d39e-4200-8ba3-654ef6f1b9c1
Note: Ensure that the "email" field is included in the SSO response from your SSO provider. If this field is empty, you may encounter errors. The email from your SSO provider will be assigned as the username in Shuffle.
KMS
Shuffle by default allows you to store authentication tokens within Shuffle itself, which are encrypted in the database. Since February 2024, we additionally support the use of external KMS systems to handle authentication, which is based on Native Actions and Schemaless. Native Actions run in the background to perform the "Get KMS key" action, and the run of the app is NOT stored.
The Shuffle KMS system is built as a third party Key:Value provider. You can reference keys from the KMS in any field marked as "Authentication" in the UI, or from within a Shuffle authentication itself (meaning you can authenticate the authentication..). The way you reference the keys is path-based, starting with kms/. Requirements:
- Have an Authentication called kms shuffle storage in Shuffle. On the Auth page, it should clearly stand out as a different type of authentication.
- The Authentication needs to be associated with an App in the IAM category
- The App needs to have an action labeled as "Get KMS key"
- If it's the FIRST translation, it may fail out without internet access to github.com
When these requirements are fullfilled, you can do the following to use the KMS system:
- Find the required parameters for the action. The first image below shows the parameters IN ORDER for Hashicorp Cloud Platform Vault.
- Use the following format:
kms/field1/field2/field3/field4/.... This NEEDS to start withkms/ Make sure the Environment is correct. It uses your default environment to connect to the KMS if not otherwise specified on the Auth page.
Example referencing the "username" in the app name "Jira":
kms/998067a9-33f2-4c4d-bbb6-4a997d784def/2e9a877f-1a89-4394-a242-f2c6d9dd2420/jira/username
If all of this is fulfilled, you can run the workflow, and Shuffle will automatically reference the KMS correctly. If it fails to authenticate, you should see a Notification show up like in the following image.
Failure handling and Debugging
During the initial configuration of KMS, there are many things that CAN go wrong. After the setup is initially completed and it works, it should not fail again, meaning spending time to set it up properly the first time is well worth it.
If you are using Shuffle onprem and have trouble with KMS translations, we suggest setting the environment variable SHUFFLE_KMS_DEBUG=true on the shuffle-backend container. This will allow you to test individual keys in ANY field, meaning you can e.g. print out the KMS value that it should translate to. This further means we will not be deleting the KMS execution, and you can see the execution on the Workflow Runtime Debugger page.. KMS debug mode is NOT enabled on shuffler.io.
If your KMS translation fails, it is most likely due to network connectivity OR translation errors for the standard in use. After running a translation, you should see the following three file categories in the admin panel:
- translation standards - The available translation standards
- translation input - The data FROM the kms used for translation. Does NOT contain values, only keys.
- translation output - The translation OUTPUT. Includes JSON paths to the correct keys
IF your translation fails, the first area to look at is the "translation output" file category, as this is where the translation for the schema happens. The hash will match the data you have sent in, which can be found in the "translation input" folder. Documentation about schemaless explains how translations happen, and how they can be fixed. If you need further help, contact support@shuffler.io
A valid KMS translation file should be in the following format:
{
"kms_value": "path.in.kms.json"
}The "kms_key" field is NOT relevant - the ONLY thing that is important is that the kms_value path is correct.
Using any KMS
KMS is supported for any system as long as the sections above are covered. It has been tested for the following:
- Microsoft Azure KMS
- Hashicorp Vault
- Google Cloud Key Management
- AWS Key Management Service
- Hashicorp Cloud Platform
- Github
- AliCloud
- ... and more! Ask if you need help.
Native Actions
Native Actions are a new way Shuffle interacts with data, built brick by brick since introducing Shuffle's Integration Layer API in late 2022. The goal of Native Actions is to enable ourselves and others to be able to perform actions towards a specific API, without necessarily know how to do it specifically for that system.
As of early 2024, this system is in active development, and we will implement features with it and help third party platforms do the same throughout the next few years.
Example usecases:
- Listing assets from your asset management system/CMDB to make a list of assets, without needing to know the Assets' API
- Blocking an Endpoint without knowing how to use the EDR API. Add this as a button to the list from the previous usecase
- Shuffle Notification Workflow: Get notifications directly to your ticketing system with minimal configuration
The Native Actions system is based on generative AI for automatic mapping of fields (Schemaless), uses Github to store configurations (Standards), and uses Shuffle's Integration Layer API to run the actions.
Inbound Webhooks
This section describes inbound webhooks to Shuffle, and how to set them up in many commonly used third-party systems. If your system support outbound Webhooks, it can also forward to Shuffle as a GET or POST request. More about webhook triggers
Wazuh
Wazuh is a SIEM platform for security operations. We've used it through their API multiple ways, but were missing an important component; alerting. That's why we've developed a simple alert forwarder from Wazuh to Shuffle.
PS: If you expect more than 10 alerts per second, you should add multiple workflows AND webhook forwarders to Wazuh
These are the steps to set it up: 1. Create a Workflow which will receive alerts 2. Add a Webhook to the Workflow 3. Configure Wazuh with the Webhook URL 4. Test the integration
1. Create a Workflow which will receive alerts This one is pretty easily explained. Go to Shuffle an make a new Workflow.
2. Add a Webhook to the workflow Add a webhook and find the Webhook URL. Remember to start the Webhook!
Copy the URL and keep it for the next steps
3. Configure Wazuh with the Webhook URL Start by logging into your Wazuh management console with access to edit the ossec.conf file. We'll first start by adding the Shuffle webhook forwarder.
- Transfer these integration files to the server. Ignore the file named "ossec.conf" for now.
- Move the custom-* files to the location /var/ossec/integrations
[root@wazuh]# pwd
/var/ossec/integrations
[root@wazuh]# ls -alh
total 76K
drwxr-x---. 2 root ossec 187 Apr 25 04:34 .
drwxr-x---. 19 root ossec 242 Dec 14 09:40 ..
-rwxr-x--- 1 ossec ossec 1.1K Dec 20 04:50 custom-shuffle
-rwxr-x--- 1 ossec ossec 4.5K Jan 19 09:32 custom-shuffle.py
-rwxr-x---. 1 root ossec 4.3K Nov 30 08:41 pagerduty
-rwxr-x---. 1 root ossec 1.1K Nov 30 08:42 slack
-rwxr-x---. 1 root ossec 3.8K Nov 30 08:41 slack.py
-rwxr-x---. 1 root ossec 1.1K Nov 30 08:42 virustotal
-rwxr-x---. 1 root ossec 6.3K Nov 30 08:41 virustotal.py- Change the files' ownership and access rights. This MAY be necessary if Wazuh doesn't have root privileges (it shouldn't)
$ chown root:ossec custom-shuffle
$ chown root:ossec custom-shuffle.py
$ chmod 750 custom-shuffle
$ chmod 750 custom-shuffle.py- Configure ossec.conf to forward to Shuffle
Go to the location /var/ossec/etc/ossec.conf (or where the ossec.conf file is). Take the information below (including
Find more fields like levels, groups and rule_id here.
<integration>
<name>custom-shuffle</name>
<level>5</level>
<hook_url>http://IP:PORT/api/v1/hooks/webhook_hookid</hook_url>
<alert_format>json</alert_format>
</integration>- Restart ossec manager. Every time you change ossec.conf, you need to restart the ossec manager.
systemctl restart wazuh-manager.serviceYou should now start seeing data sent from Wazuh into Shuffle which can be used. If data is NOT sent, make sure of a few things:
- If https in the Shuffle URL: is the certificate of Shuffle signed? Default: no.
- Check /var/ossec/logs for logs: grep -R custom-shuffle
4. Test the integration There are many ways to test the integration, but you can simplify it by setting the "level" part of the configuration to a lower number (3~), as that would trigger it in a lot of cases, including when you SSH into the Wazuh manager. After an alert is supposed to have triggered, go to Shuffle, and you'll see something like the image below.
Running custom commands with Wazuh
Active response can be used with the command "Run Command" in the Wazuh app in Shuffle. This requires an Agent (Agent list) and Command (Active response command). Below, we will show how to set up the custom command "reboot".
- Wazuh API logs will be on the manager in: /var/ossec/logs/api.log
- Wazuh agent logs can be found in /var/ossec/logs
Testing custom active response from Shuffle: The goal with this section is to set up a bash script that can run custom commands from within Shuffle.
- Log into the Wazuh agent of choice (Linux), and add the following script to the active-response folder. Give it the name "shuffle.sh". Full path: /var/ossec/active-response/bin/shuffle.sh PS: Once done, make sure it's executable: chmod +x /var/ossec/active-response/bin/shuffle.sh
#!/bin/bash
# Extra arguments
# Installing jq
if ! command -v jq &> /dev/null
then
echo "jq could not be found - installing"
sudo apt install jq -y
sudo yum install jq -y
fi
read -r INPUT_JSON
CMD=$(echo $INPUT_JSON | jq -r .parameters.alert.cmd)
CALLBACK=$(echo $INPUT_JSON | jq -r .parameters.alert.callback)
OUTPUT=$($CMD)
curl -XPOST $CALLBACK -d """$OUTPUT""" -k- Open /var/ossec/etc/shared/ar.conf and ADD the following line to the bottom of the file:
shuffle - shuffle.sh - 0- Restart the wazuh agent
/var/ossec/bin/wazuh-control restart - Log into Shuffle and import this public workflow. Make sure to change the following:
- Activate the Wazuh app if it's not already
- Add a Wazuh username & password to the HTTP node
- Add the Wazuh URL to the HTTP node, as well as Get_agents and other Wazuh nodes.
- Make sure the last block after "Get_agents" runs the command "shuffle" with the "Agent list" containing your specific agent's ID.
The default from this workflow is that it will reboot the server.
TheHive
TheHive is a case management platform for and by security professionals. One of their key capabilities is webhooks, which can send realtime updates to a third party system whenever ANYTHING is changed within TheHive (e.g. a new alert or a case task is written). Shuffle has an ideal way of handling this, outlined in this blogpost (TheHive4).
PS: There is a difference between TheHive3 and TheHive 4 on how to set this up. We are referring to TheHive4 in this section.
- Create a Workflow in Shuffle
- Add a Webhook to the Workflow
- Configure TheHive with the Webhook URL
- Test the integration
1. Create a Workflow in Shuffle This one is pretty easily explained. Go to Shuffle an make a new Workflow.
2. Add a Webhook to the workflow Add a webhook and find the Webhook URL. Remember to start the Webhook!
Copy the URL and keep it for the next steps
3. Configure TheHive with the Webhook URL Configuring TheHive is the only unique step here, and is different between versions. Full documentation about TheHive4 webhooks can be found here.
As can be seen in the documentation, there are three steps to setting up TheHive webhooks: 1. Define the webhook forwarder:
Find the application.conf file and scroll to the webhook section. If it doesn't exist, add the following:
notification.webhook.endpoints = [
{
name: Shuffle
url: "http://IP:PORT/api/v1/hooks/webhook_hookid"
version: 0
wsConfig: {}
includedTheHiveOrganisations: ["*"]
excludedTheHiveOrganisations: []
}
]Modify it to have the URL you found making the webhook in Shuffle. You can also use https by adding certificate references to wsConfig{}
- Restart TheHive
systemctl restart THeHive- Activate the webhook Run this curl command (change the URL, username and password), which activates TheHive forwarding to Shuffle. PS: Make sure you have access to the organization you want the Webhook for
curl -XPUT -u thehive_user:thehive_password -H 'Content-type: application/json' thehive_url/api/config/organisation/notification -d '
{
"value": [
{
"delegate": false,
"trigger": { "name": "AnyEvent"},
"notifier": { "name": "webhook", "endpoint": "Shuffle" }
}
]
}'4. Test the integration In TheHive UI (NOT CLI), create a new case, or add a comment to an existing case. This will then be seen within Shuffle. After the webhook is supposed to have triggered, go to Shuffle, and you'll see something like the image below.
Logzio
1. Create a Workflow which will receive alerts This one is pretty easily explained. Go to Shuffle an make a new Workflow.
2. Add a Webhook to the workflow Add a webhook and find the Webhook URL. Remember to start the Webhook!
Copy the URL and keep it for the next steps
3. Configure Logz.io forwarding After logging into app.logz.io, hover Settings Icon > Settings > click Notifications Endpoint. This will make all rules send alerts to the assigned webhook target.
Once in the notification endpoint view, click "Add Endpoint". Configure the following elements:
- Type: Custom
- Name: Shuffle (or some other identifier)
- URL: The Webhook URL from step 2
- Method: POST
- Data (should be autofilled):
{
"alert_title": "{{alert_title}}",
"alert_description": "{{alert_description}}",
"alert_severity": "{{alert_severity}}",
"alert_event_samples": "{{alert_samples}}"
}Custom data will be parsed into the field "alert_event_samples"
4. Test the integration Click "Run the test" at the bottom before saving. This allows for it to send a sample payload to Shuffle.
If the data payload is configured as in step 3, the custom data will be available as such:
$exec.alert_event_samples
MISP
MISP, short for Malware Information Sharing Platform, is one of the best Open Source alternatives for Threat Intelligence. For that reason, a lot of our users have wanted a way to handle data in realtime from MISP. What kind of data? Event updates, indicator updates, IDS flag edits, Organization edit etc.
That's why we released an extension for Shuffle which can read ZMQ messages from MISP in realtime and send them to a webhook.
Steps to set it up: 1. Enable ZMQ in MISP by going to Server Settings -> Plugins in MISP. Make sure to enable the options for forwarding Events and Attributes. 2. Install pyzmq and redis on the MISP server:
pip3 install pyzmq redis- Go to Shuffle and create a new Webhook
- Install Docker and docker-compose on a host with access to ZMQ (localhost:50000 by default). This can be on the MISP server, but at scale, it preferably shouldn't be.
- Copy the docker-compose found here and edit the environment variables to point to your ZMQ instance (PS: Don't use localhost, even if it's on the MISP server)
version: '3'
services:
zmq:
image: ghcr.io/frikky/shuffle-zmq:latest
container_name: shuffle-zmq
hostname: shuffle-zmq
environment:
- ZMQ_HOSTNAME=10.1.2.3
- ZMQ_PORT=9000
- ZMQ_FORWARD_URL=<WEBHOOK_URL>
restart: unless-stopped- Run the docker-compose!
docker-compose up -d- Test it by going to your MISP instance and changing an event or attribute
AWS S3 forwarder
Ever wanted to run an action as soon as something is uploaded to your S3 Bucket? Maybe you just want to build an S3 honeypot? If so, this part is for you. Herein we describe how you can get information about files being uploaded or changed from an S3 bucket into Shuffle.
Basic overview: User -> Upload to S3 -> S3 triggers Lambda function -> Lambda function triggers Webhook -> File is downloaded from within Shuffle
Steps to set it up: 1. Create a Workflow with Webhook inside Shuffle 2. Create a Lambda function that triggers on S3 changes 3. Configure the workflow to download the file 4. Optional: Scan file with Yara
1. Create a Workflow which will receive alerts This one is pretty easily explained. Go to Shuffle an make a new Workflow. Add a webhook and find the Webhook URL. Remember to start the Webhook!
Copy the URL and keep it for the next steps
2. Create the lambda function
After logging into your AWS, go to Lambda functions and click "Create function" in the top right corner.
Use "Author from scratch", and type in a name like "Shuffle-forwarding" and make sure to choose Runtime as Python 3.8. Click "Create function" in the bottom right corner.
Click "Add trigger" in the window, left of the function. In the next menu find "S3", before choosing the bucket you want and the "Event type". Click "Add trigger" PS: The bucket and cloud function have to be in the same location
Under Configuration > Environment variables, click "Edit". Add variable with key "SHUFFLE_WEBHOOK", and the value from step 1. Click "Save".
Time to add some code. Go to the "code" tab and paste in the code below. Click "deploy". This should now forward the request to Shuffle.
import json
import urllib.parse
import requests
import os
def lambda_handler(event, context):
# Get the object from the event and show its content type
#bucket = event['Records'][0]['s3']['bucket']['name']
webhook = os.environ.get("SHUFFLE_WEBHOOK")
if not webhook:
return "No webhook environment defined: SHUFFLE_WEBHOOK"
ret = requests.post(webhook, json=event["Records"][0])
if ret.status_code != 200:
return "Bad status code for webhook: %d" % ret.status_code
print("Status code: %d\nData: %s" % (ret.status_code, ret.text))3. Configure and test the workflow
To test the previous step, we have to upload a file to the chosen S3 bucket. If everything is connected correctly, this will trigger an event to come into Shuffle. If it doesn't, have a look at the Lambda function's logs, or contact us.
Now all we need is to do is actually download the file. This requires access rights to the bucket itself from the API in use. The app we use for this is the "AWS S3" app, with the action "Download file from bucket". See the next image for how to configure this action after authenticating it. (We may add file uploads to webhooks in the future).
And that's it! All file updates should now come into Shuffle, including a way to download the file.
4. Extra: Scan the file with Yara
In a lot of cases you would want to analyze the files somehow. But how..? Shuffle has a built in way to run Yara on files, and get results based on built-in rules (or your own). Most of the time, you would also want a response to that search, hence we've created a way to also delete the file if it matches too many Yara rules.
To do this, add in the "Yara" node with the action "Analyze file", and put "$get_s3_file.file_id" in the "File Id" field. This should match the file downloaded from the bucket.
After this, add another S3 node, and choose the action "Delete file from bucket" with these parameters:
- Bucket Name: $exec.s3.bucket.name
- Bucket Path: {{ "$exec.s3.object.key" | url_decode | replace: "+", " " }}
Last but not least, we need a way to see IF the file should be deleted. This can be done by clicking the branch between Yara and the S3 delete action > "New Condition" > with the following data:
{{ $run_yara.matches | size }} > 3
Done! Whenever a file is downloaded, it will be analyzed by Yara, checking matches, then removing the file if it's more than 3.
QRadar
QRadar send offense webhook to Shuffle.
STEP 1: Add the Script to QRadar & Config
Bash Code:
#!/bin/bash # Version 1.0.0
shuffle_url=$1 api_token=$2 offense_id=${3%.*}
auth_header="SAuthorization:$api_token"
output=$(curl --insecure -H $auth_header $shuffle_url -d "$(cat <<EOF { "offense_id": $offense_id } EOF )")
What you should do to use it:
1. Log In to QRadar;
2. Go to Admin > Custom Actions > **Define Actions**;
3. Click **Add**;
4. Fill the Basic Information (name & description);
5. In the Script Configuration set **bash** as interpreter, and import the bash script attached to this message;
6.** For the Script Parameters, add the parameters in the following order:**- Fixed Property shuffle_url - As value insert the Shuffle Webhook URI, for example https://shuffle.local/api/v1/hooks/webhook_15acc....
- Fixed Property authorization - Insert the Required headers (SAuthorization), for example in Shuffle webhook if you set SAuthorization=s873hn872n_s298ns2-98ns2ns in the Required headers you authorization value will be "s873hn872n_s298ns2-98ns2ns". If you change the token name, don't forget to change it in the bash code too.
- Network Event Property offense_id - Insert offense_id as value
7. Save
8. Deploy Changes
#### STEP 2: Create new offense alert rule
1. Go to Offenses > **Rules**;
2. Click Actions > **New Event Rule**:Rule Description
Apply Shuffle new offense alert on events which are detected by the Local systemand when the event QID is one of the following (28250369) Offense Created
Rule Responses
Execute Custom Action QRadar to Shuffle (previous added script)This Rule will be: Enabled
#### QRadar Overview
- Easy, this way each time a new offense dispatches in QRadar it will send a webhook to Shuffle containing the offense_id, then the webhook node will receive this info and pass it to the next node (QRadar App) that will perform a **get offense** ~~data~~ action using the received offense id as key.
This way you won't need to execute an api call every x time and save the last offense id. This is a better solution and improves the SLA, coz if you set 1 minute as you x time, you may have 1 minute delay or less, besides you are getting all "ungotten" offenses at once, when you can use shuffle to handle with multiple at the same time if it happen to dispatch more than one offense in QRadar at the same time or with just some seconds of difference.
### FortiSIEM
FortiSiEM is the SIEM of Fortigate. It has the possibility of notifying Shuffle through a webhook when a rule triggers, which is exatly what this documentation section is for. The main caveat: all data is XML and needs to be transformed with the Shuffle Tools "XML to JSON" formatter.
**1. Create a Workflow which will receive alerts**
This one is pretty easily explained. Go to Shuffle an make a new Workflow.
**2. Add a Webhook to the workflow**
[Add a webhook](/docs/triggers#webhook) and get the Webhook URL. Remember to start the Webhook!
Copy the URL and keep it for the next steps
**3. Configure FortiSIEM forwarding**
Log into the UI. When inside, go to ADMIN > Settings > Incident Notification. In the "Incident HTTP Notification", paste in the webhook from the previous step. Click "save", then "Test". After the test, check the workflow in Shuffle whether it triggered. If it didn't, your FortiSIEM can't access the Shuffle instance.
With the Notification Endpoint specified in the previous step, we need to decide what rules to add. By default, we add all of them. To do this, go to ADMIN > Settings > Notification Policy, and add a new policy. In here, select the "Send XML file over HTTP(S) to the destination set in...". This will make sure all alerts are sent to Shuffle.
That's it! It's now time to wait for an alert to actually trigger. When it has, make sure to send Execution Argument from the webhook ($exec) straight into an XML to JSON parser. That way you can use it easily in Shuffle.
### Splunk SIEM
Splunk is a SIEM tool for security operations. There are multiple ways to forward the Splunk alerts to external systems. Simplest way to forward splunk alerts to Shuffle is with using webhook.
**Step 1:** First we'll have to create a Shuffle workflow which will recieve alerts from Splunk. Go to /workflows in Shuffle and create a new worfklow. Then, inside workflow editor drag in the webhook from the trigger section in left pane.
**Step 2:** Click on the webhook node and then click on start. Copy the webhook url.
**Now that we have webhook running in the Shuffle, Go to your Splunk deployment server and log in.**
**Step 3:** Now we'll have to configure Splunk with the webhook URL. Once logged in, go to the **search and reporting** app and type in the query you want to create an alert for.
**Step 4:** Save the search query as an alert.
**Step 5:** Fill out all the form details for saving as an alert. At the very bottom of thr form in **Trigger action** section click on **Add actions** select Webhook.
**Step 6:** Paste in the webhook URL in URL field and click Save.
**You should now start seeing data sent from Splunk into Shuffle which can be used inside workflow for further actions.**
### Eventlog Analyzer
Note: API integration is unfortunately not supported with EventLog Analyzer. However, If you would like to forward logs from EventLog Analyzer to Shuffle. For more information please follow this [guide](https://www.manageengine.com/products/eventlog/help/StandaloneManagedServer-UserGuide/Configurations/log-forwarder.html)
### ServicePilot SIEM
ServicePilot is a high-performance analytics platform that supports observability and full-stack monitoring: metrics, traces and logs. You can collect data from many services and sources across your entire IT stack (ITIM, NPM, APM, DEM, SIEM) as well as view details of historical data stored by ServicePilot. Webhook integration service is not provided by the ServicePilot platform. Here am gonna mentioned the steps for how to use ServicePilot app workflow.
#### - What this workflow do?
This workflow is help to get all the alerts, objects, events and logs. In this workflow there are two nodes of ServicePilot app. one is for getting all the records from the ServicePilot SIEM app and another one is searching for alerts, objects, events and logs for specific entry. As a result this workflow first excecute the Get_All_Data node and after that it excecute the Get_Specific_Record node and at the last return the result.
#### - How the seaching happens?
For searching records from the ServicePilot app we need to write SQL query to search the data. we need to write query as a query parameter in the request url.
#### - Steps to create a workflow
**Step 1:** First we'll have to create a Shuffle workflow which will search for events, objects, alerts, and logs from ServicePilot. Go to /workflows in Shuffle and create a new worfklow. Then, inside workflow editor drag in the node for the ServicePilot.
**Step 2:** Click on the ServicePilot node.
**Step 3:** Change the name of the node as Get_All_Data.
**Step 4:** Write Query for searching all the datas from the ServicePilot.
**Step 5:** Inside workflow editor drag in the another node for the ServicePilot and connect it with the existing node.
**Step 6:** Change the name of the node as Get_Specific_Record.
**Step 7:** Write Query for searching specific record from the ServicePilot.
**Finally, Click the excecution button and you should now start seeing data sent from ServicePilot into Shuffle which can be used inside workflow for further actions.**
### ELK/Elastic
Intergrating Elastic with Shuffle will first require you to set up a webhook in Elastic. Below are the steps you will need to follow to ensure your alerts are forwaded into Shuffle immediately.
1. Create a new workflow in Shuffle, bottom left on your screen head to the triggers tab and drag in the webhook into your workflow. Click on it and ensure that the webhook is started. Copy the provided webhook URI and head over to Elastic.
3. Follow Elastic's documentation on setting up the webhook [here](https://www.elastic.co/guide/en/kibana/current/webhook-action-type.html).
- When setting up the webhook in Elastic you will have to provide a url to send the info to. Provide the Shuffle webhook URI you copied in step 1 above
- You do not need to provide authentication in Elastic while setting up, but just incase you do, remember to do the same in your webhook in Shuffle
4. To finish setting up your connector in Elastic you will be required to provide the body of the webhook connector, we will need to provide a JSON body for the connector. Copy and paste the below in the connector body
```{"raw": {{context.alerts}}}```
5. Go ahead and test your connector and then check your workflow executions in Shuffle if you have an incoming execution.
### Cortex
TBD: Responder executions