Assessing Organizational Risk with CloudTrail
Airplanes leave trails in the clouds to let us know where they’ve been. (flickr.com/Vicki Burton)
Recently, we’ve been experimenting with collecting CloudTrail data from Amazon Web Services (AWS). Here is a description of CloudTrail, according to the FAQ:
AWS CloudTrail is a web service that records API calls made on your account and delivers log files to your Amazon S3 bucket. …CloudTrail provides visibility into user activity by recording API calls made on your account. CloudTrail records important information about each API call, including the name of the API, the identity of the caller, the time of the API call, the request parameters, and the response elements returned by the AWS service. This information helps you to track changes made to your AWS resources and to troubleshoot operational issues. CloudTrail makes it easier to ensure compliance with internal policies and regulatory standards.
We use CloudTrail to further enhance our clients’ knowledge of organizational risk with AWS. We collect AWS information from CloudTrail, load it into our Volume Analytics product, and use the information to reinforce our risk models. While this short tutorial focuses on CloudTrail, the code can be used to read any data from S3.
Getting Started
What you need:
- Cloud Trail enabled on your AWS instance
- S3 enabled on your AWS instance
- Java AWS SDK
Collect CloudTrail Data
create an S3 bucket
configure CloudTrail
enable API access
- Finally, you can use the AWS Java API to pull CloudTrail data from S3. Make sure you configure the AwsCredentials file with the correct accessKey and secretKey. Here is my sample code for pulling data from S3 in CloudTrailTest.java on GitHub:
| //package com.volume.hooks.s3; | |
| import com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider; | |
| import com.amazonaws.regions.Region; | |
| import com.amazonaws.regions.Regions; | |
| import com.amazonaws.services.s3.AmazonS3; | |
| import com.amazonaws.services.s3.AmazonS3Client; | |
| import com.amazonaws.services.s3.model.GetObjectRequest; | |
| import com.amazonaws.services.s3.model.ListObjectsRequest; | |
| import com.amazonaws.services.s3.model.ObjectListing; | |
| import com.amazonaws.services.s3.model.S3Object; | |
| import com.amazonaws.services.s3.model.S3ObjectSummary; | |
| import com.fasterxml.jackson.databind.JsonNode; | |
| import com.fasterxml.jackson.databind.ObjectMapper; | |
| import java.io.BufferedReader; | |
| import java.io.IOException; | |
| import java.io.InputStream; | |
| import java.io.InputStreamReader; | |
| import java.io.Reader; | |
| import java.util.Iterator; | |
| import java.util.zip.GZIPInputStream; | |
| public class CloudTrailTest { | |
| public static AmazonS3 s3; | |
| public static Region usEast1; | |
| public static void main(String[] args) throws IOException, InterruptedException { | |
| while (true) { | |
| //Read credentials from AwsCredentials.properties | |
| s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider()); | |
| //Set your AWS region | |
| usEast1 = Region.getRegion(Regions.US_EAST_1); | |
| s3.setRegion(usEast1); | |
| //Name of the S3 bucket containing CloudTrail JSON | |
| ObjectListing objectListing = s3.listObjects(new ListObjectsRequest() | |
| .withBucketName(“vatraildata”)); | |
| //Iterate through all objects in the CloudTrail bucket | |
| for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) { | |
| System.out.println(“Downloading an object:” + objectSummary.getKey()); | |
| S3Object object = s3.getObject(new GetObjectRequest(“vatraildata”, objectSummary.getKey())); | |
| System.out.println(“Content-Type: ” + object.getObjectMetadata().getContentType()); | |
| //If the object contains content, treat it as a file | |
| if (objectSummary.getSize() > 0) { | |
| displayTextInputStream(object.getObjectContent()); | |
| // Optional: Delete the file after it has been read. | |
| //s3.deleteObject(“vatraildata”, object.getKey()); | |
| } | |
| } | |
| } | |
| } | |
| private static void displayTextInputStream(InputStream input) throws IOException { | |
| //All of the files are GZipped JSON | |
| GZIPInputStream gzipStream = new GZIPInputStream(input); | |
| Reader decoder = new InputStreamReader(gzipStream, “US-ASCII”); | |
| BufferedReader reader = new BufferedReader(decoder); | |
| String json = “”; | |
| while (true) { | |
| String line = reader.readLine(); | |
| json += line; | |
| if (line == null) { | |
| break; | |
| } | |
| } | |
| // Use your favorite JSON parser and go to town! | |
| ObjectMapper m = new ObjectMapper(); | |
| JsonNode rootNode = m.readTree(json); | |
| JsonNode records = rootNode.path(“Records”); | |
| Iterator recordItr = records.iterator(); | |
| while (recordItr.hasNext()) { | |
| JsonNode node = (JsonNode) recordItr.next(); | |
| JsonNode userIdentity = node.path(“userIdentity”); | |
| JsonNode accountIdNode = (JsonNode) userIdentity.path(“accountId”); | |
| System.out.println(“accountId:” + accountIdNode.asText()); | |
| JsonNode typeNode = (JsonNode) userIdentity.path(“type”); | |
| System.out.println(“type:” + typeNode.asText()); | |
| JsonNode principalNode = (JsonNode) userIdentity.path(“principalId”); | |
| System.out.println(“principalId:” + principalNode.asText()); | |
| JsonNode arnNode = (JsonNode) userIdentity.path(“arn”); | |
| System.out.println(“arn:” + arnNode.asText()); | |
| JsonNode accessKeyIdNode = (JsonNode) userIdentity.path(“accessKeyId”); | |
| System.out.println(“accessKeyId:” + accessKeyIdNode.asText()); | |
| JsonNode eventName = node.path(“eventName”); | |
| System.out.println(“event:” + eventName.asText()); | |
| JsonNode ip = node.path(“sourceIPAddress”); | |
| System.out.println(“ip:” + ip.asText()); | |
| JsonNode dateTime = node.path(“eventTime”); | |
| System.out.println(“eventTime:” + dateTime.asText()); | |
| System.out.println(“—-“); | |
| } | |
| gzipStream.close(); | |
| decoder.close(); | |
| reader.close(); | |
| } | |
| } |
Conclusion
CloudTrail is extremely helpful for gaining detailed insight into your AWS environment. As we’ve shown in this tutorial, it’s very easy to configure and pull the information into your own applications. Let us know about your experience with CloudTrail and what you discovered about your organizational risk in the comments.
To learn more about Volume Labs and Volume Integration, please follow us on Twitter @volumeint and check out our website.
Leave a Reply
Want to join the discussion?Feel free to contribute!