Assessing Organizational Risk with CloudTrail

Airplanes leave trails in the clouds to let us know where they’ve been. (flickr.com/Vicki Burton)

Recently, we’ve been experimenting with collecting CloudTrail data from Amazon Web Services (AWS). Here is a description of CloudTrail, according to the FAQ:

AWS CloudTrail is a web service that records API calls made on your account and delivers log files to your Amazon S3 bucket. …CloudTrail provides visibility into user activity by recording API calls made on your account. CloudTrail records important information about each API call, including the name of the API, the identity of the caller, the time of the API call, the request parameters, and the response elements returned by the AWS service. This information helps you to track changes made to your AWS resources and to troubleshoot operational issues. CloudTrail makes it easier to ensure compliance with internal policies and regulatory standards.
We use CloudTrail to further enhance our clients’ knowledge of organizational risk with AWS. We collect AWS information from CloudTrail, load it into our Volume Analytics product, and use the information to reinforce our risk models. While this short tutorial focuses on CloudTrail, the code can be used to read any data from S3.

Getting Started

What you need:

Cloud Trail enabled on your AWS instance
S3 enabled on your AWS instance
Java AWS SDK

Collect CloudTrail Data

create an S3 bucket

configure CloudTrail

enable API access

Finally, you can use the AWS Java API to pull CloudTrail data from S3. Make sure you configure the AwsCredentials file with the correct accessKey and secretKey. Here is my sample code for pulling data from S3 in CloudTrailTest.java on GitHub:

//package com.volume.hooks.s3;

import com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider;

import com.amazonaws.regions.Region;

import com.amazonaws.regions.Regions;

import com.amazonaws.services.s3.AmazonS3;

import com.amazonaws.services.s3.AmazonS3Client;

import com.amazonaws.services.s3.model.GetObjectRequest;

import com.amazonaws.services.s3.model.ListObjectsRequest;

import com.amazonaws.services.s3.model.ObjectListing;

import com.amazonaws.services.s3.model.S3Object;

import com.amazonaws.services.s3.model.S3ObjectSummary;

import com.fasterxml.jackson.databind.JsonNode;

import com.fasterxml.jackson.databind.ObjectMapper;

import java.io.BufferedReader;

import java.io.IOException;

import java.io.InputStream;

import java.io.InputStreamReader;

import java.io.Reader;

import java.util.Iterator;

import java.util.zip.GZIPInputStream;

public class CloudTrailTest {

public static AmazonS3 s3;

public static Region usEast1;

public static void main(String[] args) throws IOException, InterruptedException {

while (true) {

//Read credentials from AwsCredentials.properties

s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());

//Set your AWS region

usEast1 = Region.getRegion(Regions.US_EAST_1);

s3.setRegion(usEast1);

//Name of the S3 bucket containing CloudTrail JSON

ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()

.withBucketName(“vatraildata”));

//Iterate through all objects in the CloudTrail bucket

for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {

System.out.println(“Downloading an object:” + objectSummary.getKey());

S3Object object = s3.getObject(new GetObjectRequest(“vatraildata”, objectSummary.getKey()));

System.out.println(“Content-Type: ” + object.getObjectMetadata().getContentType());

//If the object contains content, treat it as a file

if (objectSummary.getSize() > 0) {

displayTextInputStream(object.getObjectContent());

// Optional: Delete the file after it has been read.

//s3.deleteObject(“vatraildata”, object.getKey());

}

}

}

}

private static void displayTextInputStream(InputStream input) throws IOException {

//All of the files are GZipped JSON

GZIPInputStream gzipStream = new GZIPInputStream(input);

Reader decoder = new InputStreamReader(gzipStream, “US-ASCII”);

BufferedReader reader = new BufferedReader(decoder);

String json = “”;

while (true) {

String line = reader.readLine();

json += line;

if (line == null) {

break;

}

}

// Use your favorite JSON parser and go to town!

ObjectMapper m = new ObjectMapper();

JsonNode rootNode = m.readTree(json);

JsonNode records = rootNode.path(“Records”);

Iterator recordItr = records.iterator();

while (recordItr.hasNext()) {

JsonNode node = (JsonNode) recordItr.next();

JsonNode userIdentity = node.path(“userIdentity”);

JsonNode accountIdNode = (JsonNode) userIdentity.path(“accountId”);

System.out.println(“accountId:” + accountIdNode.asText());

JsonNode typeNode = (JsonNode) userIdentity.path(“type”);

System.out.println(“type:” + typeNode.asText());

JsonNode principalNode = (JsonNode) userIdentity.path(“principalId”);

System.out.println(“principalId:” + principalNode.asText());

JsonNode arnNode = (JsonNode) userIdentity.path(“arn”);

System.out.println(“arn:” + arnNode.asText());

JsonNode accessKeyIdNode = (JsonNode) userIdentity.path(“accessKeyId”);

System.out.println(“accessKeyId:” + accessKeyIdNode.asText());

JsonNode eventName = node.path(“eventName”);

System.out.println(“event:” + eventName.asText());

JsonNode ip = node.path(“sourceIPAddress”);

System.out.println(“ip:” + ip.asText());

JsonNode dateTime = node.path(“eventTime”);

System.out.println(“eventTime:” + dateTime.asText());

System.out.println(“—-“);

}

gzipStream.close();

decoder.close();

reader.close();

}

}

view raw CloudTrailTest hosted with ❤ by GitHub

Conclusion

CloudTrail is extremely helpful for gaining detailed insight into your AWS environment. As we’ve shown in this tutorial, it’s very easy to configure and pull the information into your own applications. Let us know about your experience with CloudTrail and what you discovered about your organizational risk in the comments.

To learn more about Volume Labs and Volume Integration, please follow us on Twitter @volumeint and check out our website.

Assessing Organizational Risk with CloudTrail

Getting Started

Collect CloudTrail Data

Conclusion

Leave a Reply

Leave a Reply Cancel reply

Recent Blog Posts

	//package com.volume.hooks.s3;

	import com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider;
	import com.amazonaws.regions.Region;
	import com.amazonaws.regions.Regions;
	import com.amazonaws.services.s3.AmazonS3;
	import com.amazonaws.services.s3.AmazonS3Client;
	import com.amazonaws.services.s3.model.GetObjectRequest;
	import com.amazonaws.services.s3.model.ListObjectsRequest;
	import com.amazonaws.services.s3.model.ObjectListing;
	import com.amazonaws.services.s3.model.S3Object;
	import com.amazonaws.services.s3.model.S3ObjectSummary;
	import com.fasterxml.jackson.databind.JsonNode;
	import com.fasterxml.jackson.databind.ObjectMapper;
	import java.io.BufferedReader;

	import java.io.IOException;
	import java.io.InputStream;
	import java.io.InputStreamReader;
	import java.io.Reader;
	import java.util.Iterator;
	import java.util.zip.GZIPInputStream;

	public class CloudTrailTest {

	public static AmazonS3 s3;
	public static Region usEast1;

	public static void main(String[] args) throws IOException, InterruptedException {
	while (true) {

	//Read credentials from AwsCredentials.properties
	s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());

	//Set your AWS region
	usEast1 = Region.getRegion(Regions.US_EAST_1);
	s3.setRegion(usEast1);

	//Name of the S3 bucket containing CloudTrail JSON
	ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
	.withBucketName(“vatraildata”));

	//Iterate through all objects in the CloudTrail bucket
	for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
	System.out.println(“Downloading an object:” + objectSummary.getKey());

	S3Object object = s3.getObject(new GetObjectRequest(“vatraildata”, objectSummary.getKey()));
	System.out.println(“Content-Type: ” + object.getObjectMetadata().getContentType());

	//If the object contains content, treat it as a file
	if (objectSummary.getSize() > 0) {
	displayTextInputStream(object.getObjectContent());

	// Optional: Delete the file after it has been read.
	//s3.deleteObject(“vatraildata”, object.getKey());
	}

	}

	}

	}

	private static void displayTextInputStream(InputStream input) throws IOException {

	//All of the files are GZipped JSON
	GZIPInputStream gzipStream = new GZIPInputStream(input);
	Reader decoder = new InputStreamReader(gzipStream, “US-ASCII”);
	BufferedReader reader = new BufferedReader(decoder);

	String json = “”;
	while (true) {
	String line = reader.readLine();
	json += line;
	if (line == null) {
	break;
	}

	}

	// Use your favorite JSON parser and go to town!
	ObjectMapper m = new ObjectMapper();
	JsonNode rootNode = m.readTree(json);

	JsonNode records = rootNode.path(“Records”);
	Iterator recordItr = records.iterator();
	while (recordItr.hasNext()) {
	JsonNode node = (JsonNode) recordItr.next();
	JsonNode userIdentity = node.path(“userIdentity”);

	JsonNode accountIdNode = (JsonNode) userIdentity.path(“accountId”);
	System.out.println(“accountId:” + accountIdNode.asText());

	JsonNode typeNode = (JsonNode) userIdentity.path(“type”);
	System.out.println(“type:” + typeNode.asText());

	JsonNode principalNode = (JsonNode) userIdentity.path(“principalId”);
	System.out.println(“principalId:” + principalNode.asText());

	JsonNode arnNode = (JsonNode) userIdentity.path(“arn”);
	System.out.println(“arn:” + arnNode.asText());

	JsonNode accessKeyIdNode = (JsonNode) userIdentity.path(“accessKeyId”);
	System.out.println(“accessKeyId:” + accessKeyIdNode.asText());

	JsonNode eventName = node.path(“eventName”);
	System.out.println(“event:” + eventName.asText());

	JsonNode ip = node.path(“sourceIPAddress”);
	System.out.println(“ip:” + ip.asText());

	JsonNode dateTime = node.path(“eventTime”);
	System.out.println(“eventTime:” + dateTime.asText());

	System.out.println(“—-“);
	}
	gzipStream.close();
	decoder.close();
	reader.close();
	}
	}