Transcription

Copyright leCSeniorSE

ntainedinourforward- eSEC.Theforward- eatureorfuncJonalityinafuturerelease.2

thAmazonWebServices(AWS)3

#1MQ Leader4

PresentersBillBartleC SeniorSE,AWS SeaCleNateKwongSimeonYep SeniorSE,Majors SF Director,Alliances SF5

ngandAutomaJonSplunkAppforAWSDemo6

AWSEC2Infrastructure

13Worldwide Regions8

35Availability Zones9

ilabilityZone AUSWest(CA)AvailabilityZone BAvailabilityZone AAvailabilityZone CAvailabilityZone BAvailabilityZone DUSWest(OR)AvailabilityZone AGlobalRegionsAvailabilityZone BAvailabilityZone CEU(Ireland)AvailabilityZone l)AvailabilityZone BAvailabilityZone AAvailabilityZone BAvailabilityZone AAvailabilityZone BAvailabilityZone AAvailabilityZone BAvailabilityZone CGovCloud(OR)AvailabilityZone AAvailabilityZone labilityZone AAvailabilityZone BAvailabilityZone AAvailabilityZone BAsiaPacific(Sydney)AvailabilityZone AAvailabilityZone BAsiaPacific(Mumbai)AvailabilityZone AAvailabilityZone BAvailabilityZone CAvailabilityZone fAvailabilityZonesmayvary.10

yment & AdministrationApplication ServicesComputeStorageNetworkingAWS Global Infrastructure11Database

AmazonWebServicesEC2 AmazonElasJcComputeCloud(EC2) Pay- ‐as- ‐you- ‐gopricingmodel SplunkiseasilydeployedinAmazon12

BroadSetofComputeInstanceTypes nsares?llavailable

TypicalUserScenario1.2.3.4.Sign- C2instanceInstallSoTware/Splunk14

Splunk AWS CloudFormaJonReadyinminutes 15

gandsearchingLoad! GB/daySearchdrivesalargeporJonoftheload– Rarevs.Sparsevs.ReporJng– Real- ‐Jmevs.HistoricRuleofthumb–upto300GB/day– rkwelliftunedcorrectly16

InstancesInstancetype– Pricing:Spotvs.On- ‐demandvs.Reserved– urpose– GeneraJon:Currentvs.PreviousInstancesize– Workloadsize:computeunits,memory,storage– Micro,Small,Medium,Large,ExtraLarge(XL)ê MulJpleXLsizes:xlarge,2xlarge,4xlarge,8xlarge– renceserverê 50- ‐300GB/dayindexingandsearching17

ntGenhasSSDs)– GeneralPurposeinstanceshaveGBstoTBs– StorageOpJmizedinstanceshaveupto48TB!– esforusewithEC2instancesCostassociated–1TBcosts RAIDrequiredBuilt- Service– Onlinecloudstorageservice(files,data,etc )– Needthisforbackuppurposes(Snapshots)– CanalsobeusedasadatafeedforSplunk,TAavailable18

StorageBestPracJcesSingleinstancesornon- eopJmized)Instanceswillrequirestorage19

OpJon1:EBS– OpJon2:IndexReplicaJon– nshouldfactorinresiliency,use- ‐case,andcostIndexReplicaJon(IR)– – DoesnotrequireEBSforindexes– rage20

nt– EBSbackedstorageforavailability– NoreplicaJonAWSCalculatorspreadsheetavailable21

InstanceSelecJonExercise ydrivecost22

nt– chcapabilitywithSF/RFDifferences:– 5k– Increasedavailability,higherperformance23

plicaJon(IR)UsingEBSvolumes,noIR Typicallyfewerinstancesto Localephemeralstorage(SSDs)managevs.IR avolumetoanewinstance(automaJcallyormanually) erformbeCerthanEBS aforsearching IRaddsloadandrequiresmoreserversandstorage24

AmazonMachineImage(AMI) AmazonMachineImage(AMI)preferencesforSplunk– AmazonLinuxbased– BestPerformance– CostEffecJve(extra forWindows) AMIsavailablefordownload– SplunkEnterprise– Hunkê Hunk EMRbakedintoMarketplace25

BestPracJcesCustomAMIcreaJon– CreateyourownAMIusingLinuxbasedorSplunkprovided– ireroleinformaJonSecurity– SSLeverywhere privatenetwork– InstallyourowncerJficates26

Region– onitorfromoutsideoftheRegion/AZ– VPC)27

ndSizing:Load Searching IndexingIndexers(50- ‐300GB/day) c4.4xlarge16vCPU,30GBRAM d2.4xlarge16vCPU,122GBRAMSearchHeads(8 users) c4.4xlarge16vCPU,30GBRAM erver c4.xlarge4vCPU,7.5GBRAM c4.2xlarge8vCPU,15GBRAM lload.LicenseMaster c4.large c4.xlarge282vCPU,3.75GBRAM4vCPU,7.5GBRAM

Architecture&DeploymentExamples

thIndexReplicaJon30

SingleServer31

Distributed32

DistributedwithIndexReplicaJon33

pto300 GB/dayindexingwithcommonsearchloads– C4.8xlargeinstance– oncurrentusers34

F,3RF)8- s)3- ‐c3.4xlargeinstance(searchcluster)35

DeploymentCM DMC DeployerSearchHead(s)Indexer3636

elasJcblockstore) 20GB/day– c4.2xlarge(singleinstance) 100GB/day– c4.4xlarge(singleinstance) 300GB/day– c4.4xlarge– c4.8xlarge 500GB/day– c4.4xlargeasindexer(3)– c4.4xlargeassearchhead(1) 1000GB/day– c4.4xlargeasindexer(6)– c4.8xlargeassearchhead(1) 1500GB/day– c4.4xlargeasindexer(9)– c4.8xlargeassearchhead(1)37

xReplicaJon 100GB/day 1000GB/day– d2.4xlargeasindexer(6)– c4.8xlargeassearchhead(1)– c4.2xlargeasCM(1) 1500GB/day– d2.4xlargeasindexer(9)– c4.8xlargeassearchhead(1)– c4.2xlargeasCM(1)– d2.2xlargeasindexer(2)– c4.2xlargeassearchhead(1)– c4.xlargeasCM 500GB/day– d2.4xlargeasindexer(3)– c4.4xlargeassearchhead(1)– c4.xlargeasCM38

SelfHealingSplunkArchitecture39

ncesMulJpleAutoScalingPolicies– SplunkIndexers)– Performancemetrics– Timebased– ManualScaling40

ArchitectureDiagram(Splunk AWS)Search HeadinstanceSearch HeadinstanceIndexerinstanceSearch HeadinstanceAuto Scaling group – Across 3 ZonesIndexerinstanceAuto Scaling Group AZ-AAvailability Zone AIndexerinstanceIndexerinstanceAuto Scaling Group AZ-BAvailability Zone B41IndexerinstanceIndexerinstanceAuto Scaling Group AZ-CAvailability Zone CClusterMasterInstanceAutoScalingGroup of 1

ArchitectureDiagram(Splunk AWS)Search HeadinstanceSearch HeadinstanceIndexerinstanceSearch HeadinstanceAuto Scaling group – Across 3 ZonesIndexerinstanceAuto Scaling Group AZ-AAvailability Zone AIndexerinstanceIndexerinstanceAuto Scaling Group AZ-BAvailability Zone B42IndexerinstanceIndexerinstanceAuto Scaling Group AZ-CAvailability Zone CClusterMasterInstanceAutoScalingGroup of 1

ArchitectureDiagram(Splunk AWS)Search HeadinstanceSearch HeadinstanceIndexerinstanceIndexerinstanceSearch HeadinstanceAuto Scaling group – Across 3 ZonesIndexerinstanceAuto Scaling Group AZ-AAvailability Zone AIndexerinstanceIndexerinstanceAuto Scaling Group AZ-BAvailability Zone B43IndexerinstanceIndexerinstanceAuto Scaling Group AZ-CAvailability Zone CClusterMasterInstanceAutoScalingGroup of 1

tering– ReplicateacopyofyourdatatomulJplesites– Hint:AWSAvailabilityZone lityZone44

SplunkSearchHeadClusteringwithAutoScaling Auto- ‐elecJonofcaptainwithintheSearchHeadCluster es45

ArchitectureDiagram(Splunk AWS)Search HeadinstanceSearch HeadinstanceIndexerinstanceIndexerinstanceSearch HeadinstanceAuto Scaling group – Across 3 ZonesIndexerinstanceAuto Scaling Group AZ-AAvailability Zone AIndexerinstanceIndexerinstanceAuto Scaling Group AZ-BAvailability Zone B46IndexerinstanceIndexerinstanceAuto Scaling Group AZ-CAvailability Zone CClusterMasterInstanceAutoScalingGroup of 1

ArchitectureDiagram(Splunk AWS)Search HeadinstanceSearch HeadinstanceIndexerinstanceIndexerinstanceSearch HeadinstanceAuto Scaling group – Across 3 ZonesIndexerinstanceAuto Scaling Group AZ-AAvailability Zone AIndexerinstanceIndexerinstanceAuto Scaling Group AZ-BAvailability Zone B47IndexerinstanceIndexerinstanceAuto Scaling Group AZ-CAvailability Zone CClusterMasterInstanceAutoScalingGroup of 1

ArchitectureDiagram(Splunk AWS)Search HeadinstanceIndexerinstanceIndexerinstanceSearch HeadinstanceIndexerinstanceAuto Scaling Group AZ-AAvailability Zone ASearch HeadinstanceSearch HeadinstanceAuto Scaling group – Across 3 ZonesIndexerinstanceIndexerinstanceAuto Scaling Group AZ-BAvailability Zone B48IndexerinstanceIndexerinstanceAuto Scaling Group AZ-CAvailability Zone CClusterMasterInstanceAutoScalingGroup of 1

Splunk AWSFeatures FTW SelfHealingSplunkInfrastructure Jon stances49

Splunk plunkClusterMaster– hHeadClusters50

ProvisioningwithCloudFormaJon51

WhyCloudFormaJon? Fast,automated,consistentSplunkdeploymentsonAWS istraJon k s It’sfree5252

CloudFormaJonDeploymentSearchHead(s)Indexer5353

Splunk AWS CloudFormaJonReadyinminutes 54

SplunkAppforAWSDemo55

SplunkAppforAWS SplunkAdd- ‐onforAWS– – SplunkanalyzesdatafromvariousAWSsources SplunkAppforAWS– AnalyJcsanddashboardsofAWSecosystem

“CustomerslovehavingtheagilityofAWSwiththeend- ‐to- ‐endvisibilityofSplunk.”- ‐AndyJassyCEO,AWS57

SplunkAWSAppDemo58

QuesJons?59

[email protected]

am/splunk2/pdfs/technical- ‐briefs/deploying- ‐splunk- ‐enterprise- ‐on- ‐amazon- ‐web- ‐services- ‐technical- ‐brief.pdfHowAutodeskLeveragesSplunkonAWS- ‐re:Inventsession:hCps://www.youtube.com/watch?v ofYgkqK- ‐fLE61

ReferencesBlogs:– hCp://blogs.splunk.com/2012/03/07/splunk- ‐and- ‐aws- ‐sizing- ‐revisited/– hCp://blogs.splunk.com/2013/06/06/splunkit- ‐v2- ‐0- ‐2- ‐results- ‐ec2- ‐storage- ‐comparisons/– hCp://blogs.splunk.com/2013/07/31/w