2 0 1 8 S P L UNK INC [email protected] Analytics for DevOps andCloud TransformationSCITDA Leaders’ WorkshopMarch, 2018Andi Mann Chief Technology [email protected] [email protected] 2 0 1 8 S P L UNK INC .

@andimannAbout Your FacilitatorAndi Mann – Chief Technology Advocate, SplunkGlobal experience as a strategist, technologist, innovator, andcommunicator with Fortune 500 corporations, software vendors,governments, and as a leading research analyst and consultant.Business and technology commentator appearing in USA Today, NewYork Times, SkyTV, Forbes, CIO, InformationWeek, Wall StreetJournal, and more.Named to many ‘Top ’ lists including Business Insider's TopThought-Provoking Enterprise Tech Execs, Apollo Research’s TopTechnology Specialists on Twitter, Heller Search’s Top RecommendedTwitter Accounts for iT Execs, Robert Half Technology’s Top 20People Most Mentioned by IT Leaders, Huffington Post's Top 100Cloud Computing Experts, Gathering Clouds Top 5 Cloud Experts Who’s Who in Cloud, and SAP's Top 50 Cloud Computing Influencers.Published author of two books - 'Visible Ops – Private Cloud'; and'The Innovative CIO‘; blogger at 'Andi Mann – Übergeek‘; tweets [email protected] 2 0 1 8 S P L UNK INC .

@andimann 2 0 1 8 S P L UNK INC .Forward-Looking StatementsDuring the course of this presentation, we may make forward-looking statements regarding future events orthe expected performance of the company. We caution you that such statements reflect our currentexpectations and estimates based on factors currently known to us and that actual events or results coulddiffer materially. For important factors that may cause actual results to differ from those contained in ourforward-looking statements, please review our filings with the SEC.The forward-looking statements made in this presentation are being made as of the time and date of its livepresentation. If reviewed after its live presentation, this presentation may not contain current or accurateinformation. We do not assume any obligation to update any forward-looking statements we may make. Inaddition, any information about our roadmap outlines our general product direction and is subject to changeat any time without notice. It is for informational purposes only and shall not be incorporated into any contractor other commitment. Splunk undertakes no obligation either to develop the features or functionalitydescribed or to include any such feature or functionality in a future release.Splunk, Splunk , Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and regi stered trademarks of Splunk Inc. inthe United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. 2018 Splunk Inc. All rights reserved.

@andimann 2 0 1 8 S P L UNK INC .Agenda Cloud and DevOps – common elements that enable cloud and DevOps as transformative approaches Metrics that Matter – measuring cloud and DevOps for visibility into shared goals and success Analytics from planning to release – data to transform CI/CD pipelines from planning to release Analytics from release to support – data to transform monitoring, troubleshooting, & post-incident reviews Analytics for constituent insights – analyzing end user/constituent interaction for agile feedback loops Analytics for service intelligence – cross-platform data for deep insight into end-to-end constituent services Analytics for breach detection – insight into exposures, data breaches, and unauthorized user behaviors Measuring ‘the new stack’ – incl. Site Reliability Engineering’ semantic logging, telemetry, observability Advanced analytics – techniques incl. machine learning, anomaly detection, and predictive analytics Data-driven automation – coupling data with automation for actionable decisions and remediation Q&A, Wrap-up with stories in data, analytics, and transformation from Splunk, our customers, and others in the public sector

2 0 1 8 S P L UNK INC .Cloud and DevOps the common elements of people, process, andtechnology that enable cloud and DevOps astransformative approaches



@andimann 2 0 1 8 S P L UNK INC .DevOps Accelerates App Delivery VelocityCode continuouslydelivered to marketProduct Managersidentify newopportunitiesAuditorshave visibilityDevOps Teams iterate withcontinuous insightsCustomersare happy

@andimann 2 0 1 8 S P L UNK INC .Virtualization, Cloud, DevOps, Containers, MSAs,Serverless/FaaS, APIs are Disintegrating Monoliths

@andimann 2 0 1 8 S P L UNK INC .CAMS – as close to prescriptive as DevOps gets

@andimann 2 0 1 8 S P L UNK INC .CAMS – as close to prescriptive as DevOps gets

2 0 1 8 S P L UNK INC .Metrics that Matterwhat to measure in cloud and DevOps (across people,process, and technology) to provide shared goals andmeasures of success for transformation


@andimann 2 0 1 8 S P L UNK INC .I’m workingsuper hard!!That’s mystapler!

@andimann 2 0 1 8 S P L UNK INC .Yeah, but what areyou achieving?I’m gonnaneed you tocome inSunday.15

@andimannSales? 2 0 1 8 S P L UNK INC .Downloads?Installs?Users?

@andimannGartner’s DevOps ‘Metrics that Matter’Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight Tapati Bandopadhyay 2 0 1 8 S P L UNK INC .

@andimannIDC’s DevOps ‘Metrics that Matter’ 2 0 1 8 S P L UNK INC .

@andimann 2 0 1 8 S P L UNK INC .Forrester’s DevOps ‘Metrics that Matter’Velocity Business - release freq., time/cost per release, mean-time-to-change, mean-time-to-detection DevOps team - release/deploy automation %, mean-time-to-detection, mean-time-to-approvalQuality Business - MTTR, Customer experience DevOps team - Deployment failures, incident severities (by team, application, process, asset)Efficiency Business - Unplanned work, happiness of CX team with technology delivery DevOps team - Deployment frequency/duration, Incidenct severity, average provisioning timeCulture Business - Happiness with product team, DevOps team attrition, DevOps meeting frequency DevOps team - Rework rate, unplanned work, satisfaction, attrition, postmortem countForrester Research, Use Four Key Categories To Measure What Matters In Continuous Deployment, Performance Management: The Continuous Deployment P laybook, by Eveline Oehrlich and Robert Stroud, February 9, 2018

@andimannComputing UK’s ‘Metrics that Matter’Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016 2 0 1 8 S P L UNK INC .

@andimann 2 0 1 8 S P L UNK INC .More Ideas for ‘Metrics that Matter’CultureProcesse.g. Retention Satisfaction Calloutse.g. Idea-to-cash MTTR Deliver timeQualitySystemse.g. Test pass Test fail Best/worste.g. Throughput Uptime Build timesActivitye.g. Commits Tests run ReleasesImpacte.g. Signups Checkouts Turnaround

2 0 1 8 S P L UNK INC .Analytics fromplanning to releaseusing data to transform CI/CD pipelines from planning, tocode and build, testing, configuration, and release

@andimann 2 0 1 8 S P L UNK INC .Feedback Loops Enable Continuous ImprovementTest and AcceptanceApplication r, rastructureDependenciesService Levelsand KPIsMobileApplicationsCustomApplicationsCloud ServicesAPI Services

@andimann 2 0 1 8 S P L UNK INC .Getting Visibility Across Dev and mmon Data Platform – Collect, Analyze, Visualize, ShareServer, sInfrastructureApplicationsMobileApplicationsCloud ServicesCustomApplicationsAPI Services

2 0 1 8 S P L UNK INC .Metrics forResource AnalyticsInsight and prediction foreffective resource allocation Key Metrics: Work time vs. PTO/sick Hours by product/project Resource shortages Data Sources: Jira WorkDay

2 0 1 8 S P L UNK INC .Metrics forCost AnalyticsMeasurement andpredictability for cost control Key Metrics: Productive hours Labor costs Plan vs. actual Data Sources: WorkDay PeopleSoft

2 0 1 8 S P L UNK INC .Metrics forDevTeam AnalyticsInsight to coder activity forteaming & work/life balance Key Metrics: Commit count Commits by author Commit days/times Data Sources: GitHub

2 0 1 8 S P L UNK INC .Metrics forCode AnalyticsReal-time data on codequality and compliance Key Metrics: Code policy compliance Code/file/class complexity Code analysis coverage Data Sources: GitHub Sonarcube

2 0 1 8 S P L UNK INC .Metrics forBuild AnalyticsFind and fix build issues toaccelerate product lifecycle Key Metrics: Build success/failure Build queue status Build process times Data Sources: Jenkins Sonarcube

2 0 1 8 S P L UNK INC .Metrics forQuality AnalyticsAutomatically review QAresults to improve quality Key Metrics: Defects detected Test coverage Test executions Data Sources: Selenium AppScan ServiceNow

2 0 1 8 S P L UNK INC .Metrics forConfig AnalyticsMonitor provisioning/configto accelerate time to ‘done’ Key Metrics: Provisioning success/failure Provisioning times Config drift by node Data Sources: Puppet

2 0 1 8 S P L UNK INC .Metrics forRelease AnalyticsReal-time data for better,faster release decisions Key Metrics: Availability by release Tickets by release Release readiness Data Sources: ServiceNow SonarCube HP OpenView

2 0 1 8 S P L UNK INC .MEDIA & ENTERTAINMENT – APPLICATION DELIVERYImproved DevOps Agility“It’s like we were working without peripheral visionbefore and now we have it.”– Robert Gonsalves, Web Operations Key Customer Benefits Increased success rate of deployments Ability to detect issues before they affect broad production Monitoring deployment process several times per day

@andimannUse Live Data to Better Prepare For ReleaseCompare the release in dev, staging, pre-prod With the release currently in production 2 0 1 8 S P L UNK INC .

@andimann 2 0 1 8 S P L UNK INC .Analytics Across the End-to-End Software Pipeline

@andimannDon’t Forget to Measure Cultural Change e.g. Absenteeism ‘Work from home’ Staff attrition and retention eNPS Employee ‘happiness’Image source: @danslimmon - 780928 2 0 1 8 S P L UNK INC .

ONLINE SERVICES – CLOUD SOLUTIONS,APPLICATION DELIVERY 2 0 1 8 S P L UNK INC .FamilySearch Moves to ContinuousDelivery and Gains Real-Time Visibility“ Splunk Cloud has been more stable than our internalimplementation and has freed up two resources to work onsoftware development instead of managing infrastructure. It hasclearly proven to be cost-effective compared to managinginfrastructure ourselves.”– Director of Engineering, FamilySearch Successful migration from monthly releases to over 900deploys per day Ability to re-allocate 12 developers to more value-addedtasks Visibility into the AWS environment to support AWSmigration strategy

2 0 1 8 S P L UNK INC .Analytics fromrelease to postmortemusing data to transform event management, problemanalysis, troubleshooting, and post-incident reviews

@andimann 2 0 1 8 S P L UNK INC .Getting Visibility Across Dev and mmon Data Platform – Collect, Analyze, Visualize, ShareServer, sInfrastructureApplicationsMobileApplicationsCloud ServicesCustomApplicationsAPI Services

@andimann 2 0 1 8 S P L UNK INC .Data-driven Feedback Drives Continuous Improvement

@andimannGet Visibility into Ops Status and Incidents 2 0 1 8 S P L UNK INC .

@andimannAnalytics to Ensure Infrastructure Health 2 0 1 8 S P L UNK INC .

@andimann 2 0 1 8 S P L UNK INC .Analytics for Visibility into Storage and Capacity

@andimannAnalytics to Manage Cloud Resources 2 0 1 8 S P L UNK INC .

@andimann 2 0 1 8 S P L UNK INC .Source Data for Containers and MSAsData TypeContainer andmicroservices logsWhere to Find ItLogs can be ingested via any native Dockerlogging driver such as syslog, Splunk,JournalD and via Cloud integrations (e.g.,Amazon CloudWatch, Google Cloud PlatformLogging Export)Container metrics and Docker APIs (e.g., Docker inspect, Dockereventstop, Docker stats, Docker events), cloud APIs(e.g., AWS CloudWatch, Google Stackdriver)What It Can Tell YouContainer and application errors. Monitor anyperformance counters that can be calculated on top oflogs (e.g., web and application server logs)Container clusters,nodes andapplicationsDocker UCP APIs and logs from containersApplication health, nodes, clusters and containersassociated with an application, change history ofcontainers and configurationApplication logsCustom logs set by application developersWire dataWire data probes (software based)Application errors and other valuable machine datalogged by developersCommunication between an app component,application response times and payload ofapplications as they traverse your network (evenwhen you may not have direct visibility to some appcomponents)Health, performance, availability and eventsgenerated by all monitored containers