Kevin Neely's Security Notes

TL;DR

For the impatient, here is the final output from using ChatGPT to create service description documentation: – Final doc in Markdown format – (GitHub Gist) – Final doc in PDF format – (Box)

Overview

I’ve been working on building out an internal offensive security function and got to the point where I need some internal documentation as to the service(s) description, engagement model, outcomes, etc. Like a lot of planning, I started with an Xmind #MindMap, and with all the buzz around #ChatGPT, I wanted to see how well it could take what I have and build some docs for me.

In addition to Xmind, I use Obsidian for note-taking, jotting down thoughts, and organizing documentation. The object of this exercise will be to have #Markdown formatted text that I can make final edits in Obsidian and from there publish to a documentation repository.

ChatGPT Prompts

First prompt

I started with a pretty robust prompt. With the exception of the Objective paragraph at the top, this was 100% copy-paste from Xmind to the ChatGPT prompt.

Objective: I want to create a service description and engagement model for an internal red team. This service description should be formatted like a document with section headings and subheadings. Format the output in Markdown. The service components will be as follows: - the team will manage the scheduling process - the team will determine the activities that must be completed to adequately test the target - the team will determine whether a specific test should be carried out by an internal team or if an external testing firm needs to be engaged Pre-requisites for initiating a test include: - a system architecture diagram - a completed threat model document - access to the systems that will be tested The activities performed by this function will include the following items Document processes so they are Repeatable Defendable Create and perform Reporting with the following characteristics templates regular cadence tied-in to the risk review process findings go into backlogs, jiras, and project plans, depending upon the team and target demonstrated ability to remediate findings 

Engagement with developers must include 
  Regular sessions with R&D and IT teams that are developing tools and systems  
  Demonstrations 
  Threat modeling 
  
Engagement with System Operators must include 
  Educational review sessions with teams 
    system operators 
    Blue team 
    system integrators Demonstrations of exploit Collaborative remediation sessions Threat modeling

and the output

Second prompt – measurable outcomes

Now I need to define some measurable objectives of the program in #OKR format so we can measure its success. The results were pretty meh. I’ve tried OKRs and #infosec success criteria in other chats and they pretty much all come out the same. This section had to be heavily edits foor the final. Luckily, I’d already outlined them in the mindmap.

Prompts 3 thru 5 – flowchart

Now that I have a working service description and some needs-editing OKR (all a bunch of words no one will read), I need a flowchart diagram. I’m a fan of mermaid diagrams for their usefulness across various markup languages like Markdown and RestructuredText.

prompt

based upon the service description above, create a flowchart of the entire process beginning with scheduling and ending with concluding a set of tests. Output this flowchart in Mermaid syntax.

output

graph TD
A(Schedule Test)
B(Review Prerequisites)
C(Plan Activities)
D(Determine Testing Resources)
E(Perform Testing)
F(Record Findings)
G(Produce Report)
H(Engage with Developers and System Operators)
I(Conclude Tests)

A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
H --> E
H --> I


In this flowchart, the process begins with scheduling a test (Step A), followed by reviewing the prerequisites for the test (Step B). The team then plans the activities required for the test (Step C) and determines whether internal or external testing resources are required (Step D). The testing is then performed (Step E) and the findings are recorded (Step F). The team then produces a report (Step G) and engages with the developers and system operators (Step H) to remediate any issues. The process concludes with the finalization of the tests (Step I).

This was as decent start, and I had a couple iterations:

1. adding a decision add a decision to the chart after step E in order to determine whether the completed tests are adequate to move on to testing completion
2. inserting some more steps In the Mermaid flowchart above, tests need to be concluded prior to producing a report. Also, add a step called "Send to Product Management and Security Leadership" occuring in parallel with the "Engage with Developers and System Operators". Also, conclude with a step called "apply recommendations to target systems"

There were still some errors at this point, such as ChatGPT insisting on putting the “Conclude Tests” step at or very near the end, even though it’s really about the mid-point of the workflow. I could have continued the prompts, but at this point I decided to finish this off by hand.

Prompts 6 and 7 – RACI

Now that the processes are defined, we need to identify the responsibilities of each of the roles and how they interact.

From this, it’s clear that ChatGPT is good at formatting and generating a RACI matrix in #Markdown format, and this version has reasonable values for the roles, even if I don’t agree with them. Still, it was useful enough to have a good foundation that I could tweak, but before that, I added one last parameter:

add a new role to the matrix called Red Team Manager. This role should be responsible for teh scheduling and accountable for everything else. Add two new processes called Review Findings and Remediate Findings. The Developer is responsible for the former and the System Owner is responsible for the latter. Recreate the RACI matrix with these new parameters and output the Markdown code.

And this changed the RACI to basically make the manager accountable for everything.

Finishing up

At this point, I felt like I had the elements I needed, so I began the process of copy-pasting them from the interface into Obsidian and making tweaks to get a usable service description document.

The final output from using ChatGPT to create service description documentation: – Final doc in Markdown format – (GitHub Gist) – Final doc in PDF format – (Box)