Speaker 1: 0:22

reporting from Knowledge 2024, the annual ServiceNow conference. Joining us is Pat Casey. He's Chief Technology Officer and Executive Vice President of DevOps at ServiceNow. We're going to discuss the effect of AI on platforms like ServiceNow in more detail. Welcome, pat, thank you for joining us.

Speaker 2: 0:39

Thank you, it's a pleasure to be here.

Speaker 1: 0:42

ServiceNow is fully committed on AI. If I look at what I've seen this week, what does that mean for organizations?

Speaker 2: 0:49

In terms of org structure, or are you thinking more in terms of just focus on what we're working on, or both?

Speaker 1: 0:54

Yeah, both basically.

Speaker 2: 0:56

So structurally, we actually did well. Servicenow used to have a standalone AI team and we had a combination of research, we had applied science and we had engineering and we built our AI solutions within that structure and it worked pretty well. We got a lot of good product out. But what we've seen in the last call it 18 months is that AI is no longer in adjacency to our platform. It is a fundamental part of the core platform that we do. So we reorganized and we actually built a unified technology group. Now it's called the Plato Organization, so it's platform and it's applied technologies and it's AI all bundled together into one organization, because we wanted to actually get the org structure aligned with our vision. And our vision is that the platform is going to have AI woven throughout it so that it is not ever an extra or an adjacent or a sidecar piece of technology.

Speaker 1: 1:57

Okay, and what does that mean for your customers?

Speaker 2: 2:00

For our customers. It means they should get really good AI solutions that are fully integrated with the workflows they already have. So if you look at ServiceNow, we're a single platform technology stack and that's pretty unusual in the enterprise software space. You know most of the people out there are, you know, portfolio companies they have. If you're Microsoft, it's a great problem to have, but you've got Outlook great product. You've got Dynamics good product. You've got PowerApps. You've got a bunch of stuff out there.

Speaker 2: 2:30

So if you want to add technology or AI to those various technologies, you pretty much need to build an AI co-pilot, something that sits outside of your core technology stacks, that can span across them. We sort of have the advantage that we don't have to take that approach. We can build technology AI right into our core platform because all of our apps use that one specific platform. By doing so, we expose it natively inside all the applications and just you know, trivially you don't have to go to some other tool to interact with the AI. It's a core part of the existing workflows. It's a button on an existing form, it's a step, it's an extra better response from the chatbot. It's different or better responses when you do the search. So it's allowed us to weave that AI experience into the technology stack, into our customers' applications, really seamlessly.

Speaker 1: 3:21

Okay, but what I've seen so far is that you weaved it in to your applications that you have built on top of your platform. I'm wondering if customers have built applications on top of your platform, can they add AI as well?

Speaker 2: 3:34

Absolutely so. The platform actually has a feature called the Generative AI Controller or Gen AI Controller. It's an actual utility inside the platform where you can invoke various large language models and you can invoke any of the models we have in our farm. You can invoke OpenAI. You can actually invoke there's a German model, I think Aleph. Alpha is linked in there. I think there may be a couple others we've added as well.

Speaker 2: 4:02

So, as a customer, you can build a ground-up AI solution on ServiceNow and just directly call the GenAI controller. You can also take our existing out-of-box flows and prompts and I think starting in May, you had the ability to actually tweak those prompts and you can even swap out the model that our out-of-box workflows are using in a lot of cases. So as a customer, you've got a lot more power and control over that. You know, transparently, 90% of my customer base will probably never do that because they're quite happy. Just hey, I bought the product, I want it to work, but there's going to be some really sophisticated customers who want to do that and I've even seen some customers do the opposite, Like I know a big. It's a big German firm that had an in-house solution they built on top of ChatGPT. It was a ChatGPT before and they actually ended up moving it into ServiceNow to invoke our large language models because they could get better response times and similar quality outcomes there. So that open framework we have has really helped us make our customers successful.

Speaker 1: 5:04

And which role does data play into your AI vision?

Speaker 2: 5:09

It's a good question. So if you look at the models, there's two places where you need a lot of data. One is training. We do train some of our own models. We've released a couple in open source, like StarCoder. I think we said StarCoder 2, the new, improved StarCoder. We also have some in-house models that we haven't released that are we trained them to do special stuff. In those cases, having training data is super useful. But it turns out that you don't need as much domain-specific data as you might think. You know. We have plenty of that the primary data. Where you need, you know, petabytes of information to train, like a core base model. That's pretty conserved.

Speaker 2: 5:49

Most people in the industry use some well-understood data sets. There's a books data set. There's a God help us. There's a Reddit data set people use. There's something called Common Crawl.

Speaker 2: 6:00

Where data really comes in is when you're trying to run those models and build your prompts is when you're trying to run those models and build your prompts, and the kind of gold standard for a lot of AI interactions is that you take data that's local to the customer. You bundle up a prompt and you send that prompt to a model and the model is not customer-specific, it's the same model everybody uses. So the example we always use is a knowledge base. If you go to, like Siemens is a customer service. Now, if you go to Siemens' knowledge base and you type in what's my paternity leave policy and you hit enter, what we actually do behind the scenes is we query directly in Siemens' knowledge base, which is in service now. We find the five best documents that we think might have the answers the traditional search. We then bundle up those five documents and we send them to the large language model and we say, hey, does the answer exist in one of these five documents? If so, please summarize a response for me.

Speaker 1: 6:57

And then the model comes.

Speaker 2: 6:58

Excuse me, the model comes back and gives the answer. So to do that that's kind of called RAG you need to have access to the knowledge base in the first place. Strangely, the model doesn't need to be trained on it, though, and that's one of the powers of the new Gen AI models.

Speaker 1: 7:15

The example you just gave is when you have that paternity information within ServiceNow. But ServiceNow is also a platform that connects down to I don't know Salesforce, sap, workday and many, many other SaaS applications. What if the data is in?

Speaker 2: 7:27

there. Well, you've got two options. One option is, of course, you can replicate it into service now People have been doing that for years. But the other thing I think you saw some of this on stage we are making big investments on making it easier to connect to third-party data sources, in which case we will actually expand our search into that third-party source, like, in which case we will actually expand our search into that third-party source, like on stage, we use SharePoint as an example. Your prompt comes in. We do query the local knowledge base. We will also query SharePoint and we will merge those two document sets together and then we will send it to the large language model.

Speaker 1: 8:02

And will you do that with Integration Hub or with a different product?

Speaker 2: 8:05

It is done through Integration Hub. Integration Hub has a set of things called connectors, so we're making a big investment in the connectors there and we're really focusing on trying to make them easier to use. The feedback we've gotten from customers in the past is that it's harder than they want to do that sort of thing, so it'll make it easier.

Speaker 1: 8:24

And we'll also be looking at catching the data that's in SaaS services, because I can imagine that SAP and Salesforce will give you the answer pretty quickly, but there are maybe smaller SaaS solutions that can't handle all the API calls that the platform of ServiceNow is sending.

Speaker 2: 8:44

I think we're going to have to cross that bridge when we come to it. Right now, service now does have a capability to cache things, but my understanding is right now we are sending the prompts through directly, and part of that's due to the security models, because when I search, if you do a query, I, from a security standpoint, I should interact with sharepoint as you, and then if you know one of colleagues does a query, even if they do the same query, I should interact with SharePoint as them, because perhaps they have access to different things there. So it makes the caching a bit more complicated. With the enterprise security models on the back end Make sense.

Speaker 1: 9:22

It makes sense. Definitely it could also be a strategy to just add a data warehouse to the platform and run more locally. Is that an idea that's being brought up?

Speaker 2: 9:34

Well, actually that's our traditional approach. We've told people hey, servicenow, we're not a data lake, but you can put a lot of data in ServiceNow and knowledge bases are small, so just replicate your knowledge base into ServiceNow and we'll take care of it. Lots of customers do that. But one of the things we've heard recently is A it's a pain in the butt and B we're just adding more and more external knowledges, so can't you just search them natively? So we're sort of trying to align with where the customers want to be on this. But today and last year and three years ago, you 100% could replicate a third party knowledge base into ServiceNow and a lot of people did.

Speaker 1: 10:12

Okay, but to make it clear for the listener, searching external data sources is possible, but it needs to be a bit better and that's on a future roadmap somewhere.

Speaker 2: 10:25

Yes, it actually should be later this summer.

Speaker 1: 10:27

Oh, later this summer it's really quite close.

Speaker 2: 10:30

The demo you saw on stage was not a fake demo. That was real working code, okay.

Speaker 1: 10:35

It's good to know you also introduced a lot of AI features for your own tooling. As it may be, you have the service catalog generator and you have the. What was the other one? Oh yeah, the workflow builder Playbook generator.

Speaker 2: 10:51

Playbook generator yeah.

Speaker 1: 10:53

You're all adding AI to that to make it faster. How much time or effort do you think it will save your customers to build those?

Speaker 2: 11:05

I think it depends by the use case. You know, the one that we started with was text to code, and then the engineers. You know you do a little prompt, it builds a code snippet for you. It tends to make the act of coding a lot more efficient, you know, and it varies. We call it 10 to 40% of the numbers we've seen. But if you're a professional engineer you don't spend 100% of your time coding. You spend, you know, some time doing testing. You spend some of your time in design meetings. So it's hard to know, like, what the actual benefit is to overall engineering productivity. But that act of coding you call it 10 to 40% is what we've seen've seen what proportion of your time you spend on coding varies by engineer. For some of the things like catalog generation, it's a bigger proportional benefit, just because catalog generation can be pretty tedious. So those benefits we think are going to be bigger. But that's new enough. I don't have empirical numbers that I can share on that one.

Speaker 2: 12:03

But the playbook generation is yeah, I think the so 100. We do think it's going to save people time. I think the other thing we're hoping it's going to do and we do expect it'll do is it makes it easier for people who don't have as much experience with the platform to do some tasks in the first place. So we're hoping that the text to playbook will take a bunch of people who today probably can't make a playbook and like oh, with this it builds a playbook for me and then I can modify it a little bit and then I'm good. So we're hoping to expand the aperture and get more people successful building on the platform and make everybody more productive. It's an and Make sense.

Speaker 1: 12:45

It makes sense. I was just thinking because, if you're going to add more data sources and you also have the POM builder, it's not really clear to me Is the POM builder active within the whole platform or is that limited to IT service management at this moment?

Speaker 2: 13:02

The platform capability for customizable prompts is going to be platform-wide, but it has to be linked into the applications. So the first application we're linking it into is ITSM, because that's our single biggest product line. I think it's probably $5 billion of revenue or something like that, but we do expect to add that to any place where you're doing a prompt. It should be going through the customizational framework, probably by end of this year, maybe by early next year.

Speaker 1: 13:28

And your partners in ISVs that are building applications can also link to PromBuilder through their products.

Speaker 2: 13:33

Absolutely yeah. The idea is we're a single platform architecture and we don't cheat. We build applications the same way third parties do. There's a little bit of a benefit because if we get really stuck, we can always ask the platform team to add a new platform feature. But when that platform feature comes out it's available to everybody. There aren't any special platform features that only ServiceNow can call.

Speaker 1: 13:58

Yeah, and how do you look at prompt engineering within the platform, because that's a skill on its own, basically.

Speaker 2: 14:05

It is. So it is a skill. For the readers who don't know, it's a little black magic-y because you're interacting with an underlying large language model that is somewhat non-deterministic in its behaviors, and so being good at prompt engineering is not necessarily the same thing as being a good engineer or even being a good data scientist. We have found people who are good prompt engineers are. It's a breed unto itself. So we now have people inside service, now that that's their job. They're good prompt engineers.

Speaker 2: 14:41

My advice to your readers would be don't assume, because it looks easy, like on a YouTube video, that it really is easy. It's pretty hard. So when I expose a framework like that, you may well have some of those people inside your company be able to build great prompts, but expect that it's a specialized skill. Right now, the people we've seen who are best at it often have come out of our applied science group, but not exclusively. We've had some decent success. One of our designers turned out to be really good at it. So just having that mental thought process, where you're, how can I outsmart this model? It's a sort of a unique human skill.

Speaker 1: 15:21

It's a sort of a unique human skill. And how do you plan to bring prompt engineering to your customers then, because you're going to open up the prompt builder in the near future and they're going to work with it. They're probably going to look at ServiceNow for a bit of help.

Speaker 2: 15:37

Yeah. So there's two things we're going to hopefully help them do. Most of the time they're probably going to start with a prompt we wrote, which we hope is a pretty good prompt. It's usually easier to take something that's working and modify it a little bit than to start from the ground up. The second thing we want to do is there's some certain core.

Speaker 2: 15:55

If you think about something else we announced on stage called the knowledge graph, one of the core use cases for the knowledge graph is to help with prompt engineering, because if it's a prompt about a person, for example, you probably want a ball of information about that particular human being to send to the model.

Speaker 2: 16:12

You know you should know their job, how long they've been with the company, are they married or not, do they have health insurance, what corporate assets do they have. You know all that stuff Like what's their job. You bundle all that together and you send that along with your prompt. The model is going to build a better response tuned to you. If you work on the factory floor, it's probably going to give you a different response than if you're an accountant. The ability to bundle up that information about you is something that Knowledge Graph is really good at, and those APIs are exposed directly inside a prompt builder, so you don't have to figure that out. You just say include the user graph or include the hardware graph or the location graph, and that gets exposed right inside there.

Speaker 1: 16:55

Yeah, so they don't have to bring all the data themselves.

Speaker 2: 16:57

Yeah, I mean theoretically you could. But with the Knowledge Graph feature and the user graph feature, you just say include this graph and then behind the scenes we take care of it for you. Okay.

Speaker 1: 17:08

And is a knowledge graph also going to be extendable or something to?

Speaker 2: 17:11

third-party sources by third parties or by customers. Okay, it's probably similar I mentioned. I think most customers will probably use our prompts, as is but, some will change it.

Speaker 2: 17:22

I think same thing with the knowledge graph. I think most customers will use our predefined graphs, but the framework is extensible. So the idea being that a customer, you could make a new graph or you could define, you could change the definition of a graph we had, Like you could add things to the user graph or remove things, or you could say hey, in EMEA, I'm not allowed to include your job history in the graph. Maybe it's a privacy concern.

Speaker 1: 17:50

Listening to all this, I'm wondering how far along is service now with its AI vision, because what I'm getting from you is there's a lot coming as well. So are we just in the early phases, or do you think you already set a major step?

Speaker 2: 18:03

I guess the answer would be both. I think we are in very early days of a very real technology shift. Um, and I felt this way, um, when netscape hit, like when the browser hit. I felt a little bit of this way when the SaaS revolution started rolling by, and what you saw in both those cases is there's this sort of like frantic adaption period early on where everyone's rushing to try to apply the new technology to the existing paradigms, and then there's a settling in period and then there's some new paradigms sort of start to arise.

Speaker 2: 18:42

I'd say we as an industry are still sort of in that adaption phase where, when you look at most AI, it's largely about how can I apply AI to existing paradigms? I think in that model I think ServiceNow we're fairly ahead of the curve. We have the advantage that we had AI researchers. We knew how large language models worked. You know we had a good expertise there before this became popular. But I fully believe if you look, if it's 10 years from now and you look back at this revolution, we will look back at 2024 and say, man, you know that was. We were missing three key parts of AI enablement. We just don't know what those are yet.

Speaker 1: 19:21

Yeah, so there's a lot more to come.

Speaker 2: 19:24

A hundred percent. The cool thing that I think, though, is last knowledge. We talked about AI. I don't know if you were here for that, but it was technology. We stood on stage and we showed you technology, and it was cool.

Speaker 2: 19:40

This year, it's products. It's products with customers who have used it and gone live and can tell you it works, and these are the benefits we got. So, even inside of 12 months, this has gone from sort of a gleam in the eye of the researchers and the technologists into real live shipping products and the the rate of innovation that it's not just us industry-wide the rate of innovation here. It's fun.

Speaker 2: 20:02

It is really a fascinating time to be in the technology industry, because we you know technology goes through sprints and drifts, you know, and we are in a sprint right now where there's lots of fun stuff changing, there's lots of new applications rolling out, and that's, I think, more interesting than living in the period of time where it's like, hey, we're making things a little bit better every year. There's a lot of things we're changing now, and the reality is industry-wide. We will make some false starts, too, we will rush some stuff out. That's not good, and we will apologize to our customers and move on, but we will also rush some stuff out which is really good on, but we also rush some stuff out which is really good.

Speaker 1: 20:41

I think it's really hard to create a strategy forward because you have to change your strategy maybe every three months because new things come out and new other big tech vendors come out with a new LLM that you're like oh wait, this is interesting. Maybe we need to shift our focus a bit.

Speaker 2: 20:57

You saw that If you looked at ServiceNow, we were pretty structured. We released twice a year and there were some business units which went quarterly, but twice a year we're back to four times a year. We're back to quarterly. We've ramped up our iteration cadence because we need to keep up with, we've got to get product out to customers. And you're absolutely right on the models as well. You know, this time last year it was, it was chat GPT, maybe Lama 2 was just starting to come out. You know, right now it's Mixtral or Mixtral or Mixtral depending. Lama 3 is just coming out. I'm sure six months from now there'll be another new cool open source model. So you got to be flexible on the backend and that is one of the nice things that, you know, having our own research team helps us do, because they can help evaluate these things.

Speaker 1: 21:45

Switching back to what you all announced this week. Is there a specific product you're very excited about?

Speaker 2: 21:51

There's a bunch of things I'm really excited about, I think generally on Gen AI, my personal conviction. I mean there's three things you do with Gen AI, oversimplifying a lot. You deflect calls Like I want to help the customer get the answers they want without ever involving a human agent that deflection. Second, I want the agent more productive, like that's 100% very valuable. And then the third thing is we want to make it easier to actually deploy these things in the first place, make the developer, the administration, make the active building things more efficient. I think the last one because I tend to talk to a lot of practitioners and technologists. We all care about that, so it gets a lot of airtime Agents. I mean that tends to be the way that people think about service delivery that tends to get a lot of airtime.

Speaker 2: 22:40

I remain convicted people are undervaluing the deflection aspect of that, because any time I can answer your question without involving a human agent, that agent was infinitely productive. You got a better answer faster. So as a consumer, you got a better outcome and the agent had to do literally nothing. It is the best of all possible worlds and when I talk to some of our big customers, they are getting real measurable improvements in deflection out of this and I run our support group.

Speaker 2: 23:11

My deflection pre-Gen AI was at 11% and I remember running a two-year project and we were excited because we got to 14%. A 3% bump was a two-year project and like we were excited because we got to 14%, like a 3% bump was like a two-year project. I'm seeing people like double their deflection rate with the Gen AI stuff just basically the knowledge Q&A. So I really do remain. I'm a firm believer. That is the hidden is the wrong word, but that's the thing I always recommend customers look at Like really look at the deflection, because you might get better mileage out of that than agent productivity. You should do both, but if you have to start somewhere, I always say start with deflection.

Speaker 1: 23:47

Yeah, that's definitely a good point. We visit a lot of these conferences and it's a lot about Gen AI now. So one of the questions I always ask is because if we take Gen AI out of the equation, your platform also received a bunch of new features. Yep, but there wasn't talked about because Gen AI took the spot. Are there any new things without Gen AI that you think we didn't talk about it, but it's really really yeah, well, I did talk about on stage, like the Raptor DB it's.

Speaker 2: 24:17

It's really pretty cool. Um, I didn't have time on the stage to really give the details, but it's a whole new database we wrote.

Speaker 1: 24:24

It's a whole new database.

Speaker 2: 24:26

It's based on the Postgres code base, but we also bought a German company called Swarm64 out of Berlin.

Speaker 1: 24:32

Because previously you were using MariaDB right.

Speaker 2: 24:34

We were, we still do we still do. But we are in the process of migrating a lot of the customer base. I don't think I'll be done migrating until 2026 but I've got over 500 customers on raptor already. Okay, um, and it's a. It's in-house developed, it's postgres, it's a company called swarm and it's a bunch of, frankly, code we wrote, but it's super optimized for our workload and I I mentioned this about three times the throughput. That was the onstage For most workloads.

Speaker 1: 25:03

It's also faster, but there just wasn't time to talk about both onstage, but it's basically a fork of Postgres with technology you guys bought from the company.

Speaker 2: 25:13

Yeah, I don't know if you know your databases, but Postgres is good OLTP For OLAP workloads. It's okay. We bought a column store from Swarm64, and then we actually it's a hybrid, so it's actually got the column store running alongside the underlying OLTP database and it's really good at both of them. And there's a bunch of patented stuff in there too. It's kind of slick behind the scenes, but usually column stores you can't update them fast enough to keep them in sync with a transactional database. With the Swarm technology we can. We can keep the column store transparently aligned.

Speaker 1: 25:55

And I have to ask is this move just for performance reasons, or do you also foresee new features that you can now offer because you have a maybe better database?

Speaker 2: 26:04

The driving force was scale actually, Because, like, MariaDB is a good technology, Like I, like those guys Monty's a good guy, I mean, I know a lot of them but it was also built like back when a computer had like one CPU. You know the modern chips have so got off. They have huge numbers of cores Like I think I have 112 cores in my database servers now and Postgres just uses those high core count computers really well. Just the Postgres architecture and the swarm architecture uses that really well, so I can really take advantage of modern hardware with it. And the scaling was the driving force. It is also faster. I wanted to make sure that I was giving people better reports, better analytics, and there are some cool things we can also do with, like graph traversal and other things there that are present in the Postgres engine that are not as well exposed inside Maria. I don't want to say anything bad about Maria. They are good human beings and people should still feel fully confident to use Maria.

Speaker 1: 27:01

Yeah, but sometimes you have different needs.

Speaker 2: 27:04

Our customer base has gotten bigger too, Okay.

Speaker 1: 27:08

I want to thank you for your technical knowledge and insight into the Now platform. I hope to talk to you again soon.

Speaker 2: 27:16

Absolutely, it's a pleasure. Thank you, thank you.