Lessons Learned from 30 Days of Company-Wide AI

What lessons has the Knownwell team learned from just 30 days of company-wide access to ChatGPT?

And what led to David’s stroke of the pen decision to give everyone access to the paid version?

And what does a smart rollout strategy look like if you’re considering your own enterprise deployment of AI tools?

Hi, I’m Courtney Baker, and this is AI Knowhow from Knownwell, helping you re-imagine your business in the AI era.

As always, I’m joined by Knownwell CEO, David DeWolf, Chief Product and Technology Officer, Mohan Rao, and NordLite CEO, Pete Buer.

We also have the second part of Pete Buer’s discussion with Ardy Tripathy of OpsCanvas about using AI to automate infrastructure at scale and more.

But first, if all the latest LLMs out there can’t reason well enough to solve a simple logic puzzle, should we really be trusting it with far more complex business problems?

Pete Buer joins us, as always, to break down the business impact of some of the latest and greatest AI news.

Hey Pete, how are you?

I’m good Courtney, how are you?

I’m doing well.

A new research paper from Apple is causing waves in the AI world by seriously challenging the idea that today’s leading LLMs can reason effectively.

Pete, what’s the takeaway here?

So the Apple paper highlights what it deems to be central flaws in assumptions around generative AI, that the models may look smart on casual use, but when complexity increases, they often collapse.

I don’t know if you’ve seen it, but there’s another story floating around right now that sort of condemns generative AI in the same way.

It’s about the 1979 Atari chess console.

Have you seen this?

I know.

No, OK.

So during Jimmy Carter administration, the Atari game, when it came out, one of the first applications was playing chess.

And this fellow played the Atari console against ChatGPT 4.0.

And it won every time.

ChatGPT couldn’t compete even in easy mode, kept mistaking rooks for bishops, and so on and so forth.

I’m personally inclined not to give in on the sort of anti-hype or the negative hype around this right now, because we have to remember, these are tools, right?

And like any tool, generative AI has got stuff that it’s kind of more great at and stuff that it’s less great at.

It’s great at synthesis and making decent decisions with the right contextual data on board.

It’s less great at making decisions without the right context and downright lousy at times when there’s novelty, anomaly, disruption in the data set or the task set.

This is, at the end of the day, where the human belongs, right?

This is the coming domain of the human in the mix.

And as we know, the most powerful applications of AI are the ones where humans and technology are working together to accomplish something special.

So all of this to me just feels like kind of a stage gate, a status check on the performance of this tool.

And it’s all valuable input for where the development roadmap ought to go next.

We know what’s working, what’s not working, and what we need to improve, and let’s keep working on it.

I do think in all of this, if I’m not willing to take the bait on hype, that there is a lesson for business leaders that were a little bit lazy about how we think about the right applications of AI.

Too many leaders see AI as just a magic wand that you wave over a set of tasks, and they’re instantly transformed into something that’s digital and better.

Let’s make sure we understand the limitations of generative AI as we’re looking at our transformation possibilities, and make sure we understand the contextual data and how much of it there needs to be in order for the application to succeed.

And also make sure you’re making calls based on that, about where AI belongs, where AI should be working with humans, and where AI doesn’t have a place in the lineup.

Well very interesting, Pete.

Thank you as always for keeping us in the loop.

Thank you, Courtney.

During our recent company retreat, we made a decision that had been months in the making, even if we didn’t realize it.

Giving everyone in the company a paid license to ChatGPT.

One month in, how’s that working out for us?

I sat down with David and Mohan to break it down.

David, Mohan, welcome back.

We’ve talked a lot about how AI can empower individuals and teams.

But here’s a question more companies are wrestling with now.

What happens when you roll it out to your whole organization?

This actually came up, this episode idea came up because we just made a decision on how we were rolling it out to our entire team for everybody to have access to ChatGPT.

So in this episode, I thought it would be really helpful for us to break down our thinking and our thought process on the why, the how, and the lessons we’ve learned so far about operationalizing generative AI across the business.

So we’re having this podcast because I think most people would assume right out of the gate, we had chosen a platform, rolled it out to the whole company, and I think there’s something to be said for just being vulnerable and open to the process that we went through and what it was that ultimately helped us kind of tip the scales and land on utilizing ChatGPT at an enterprise level across the organization.

Yeah.

I think I’d break that down actually in two different, very specific points of time.

The first one is, when we first started up, we made a deliberate decision that we weren’t going to standardize on a single platform because we wanted to experiment.

Remember, this was early on.

This was September of 23 that the three of us first started, and then we’ve organically grown one person after the next, after the next, up to 17 people now.

But had intentionally in those first several months, said the market is too young, it’s too immature, we need to explore, we need to experiment.

There’s several of us who have had subscriptions, but we all have been doing it on our own as we play with different things.

And obviously, our engineering team has even more tools and different tools, and we’ve all had our own use cases.

So it’s fairly intentional early, but honestly, I don’t think we ever revisited it after that point in time.

We just promoted, use it, experiment, figure it out.

And I think it came to a head.

We had in all hands, we get our team together twice a year.

And it just leading up to that, and then at that event especially, I think it became very obvious that there were a lot of people that knew how to experiment and knew how to do it.

There are a lot that weren’t as comfortable and weren’t just going picking something and doing things with it.

And I think we started to really remind people over and over, right?

There’s been this big burst in the last three, four months out in social about, you know, it’s time.

If you’re not using it, you’re behind.

And we’ve been on that bandwagon telling people, let’s get leverage.

How do we become 10xers by leveraging this technology?

Let’s not just talk about it.

Let’s not just use it in engineering and in our product.

Let’s use it for our own work and challenge the status quo of how we work.

And as we push that harder and harder, it just became obvious that we needed to support everybody with no, just a default subscription of here you go.

Everybody in the company is going to have the enterprise access to ChatGPT.

In addition to that, we’re still open to other tools, et cetera, et cetera.

But everybody’s at least going to have that barrier of entry starting point to leverage.

I think too, we should also differentiate between our product, which is very different than what we’re talking about now, which is really kind of what we use in the administration of the business, you know, across the board.

And I would say for me, personally, it was really making sure that we had the security, the way that we needed it across the enterprise for everybody in the company that was using it.

So on that note, Mohan, what were the things for you that kind of pushed it over the edge to be really clear that we needed to move towards ChatGPT across the enterprise?

I think, you know, every employee is different, right?

Some need permission and some don’t need permission, right?

So the ones that didn’t need were just innovating on their own, which we knew about and which we encouraged, in many cases, paid for it as well, just because you need to let a thousand flowers bloom before you kind of figure out what garden you want to build, right?

So we were in that process of figuring it out.

Also, I think we started getting to a point where it was more, it was going to be more efficient to have a team subscription, I mean, ChatGPT for a team, because what that allows you to do is figure out the security model, that it is, there’s a single sign on, it’s centrally, administratively managed in terms of how you give out these seeds, as well as you can figure out what are the enterprise controls that ChatGPT offers, that you can set for your company.

So this way, data does not leak out, so on and so forth.

So these were very important, along with just making sure that everybody understands that, you know, just in case there’s some employee out there, where, you know, the work that you do is the work you do, whether you use the tool or not, you might as well use the tool, right?

You give them the permission to say, you don’t need to be a secret employee, you’re an open AI employee, right?

So, so it was just sort of just all of these just came together.

Anyway, there, everybody, almost everyone was using something, and it was time to make it official, and get to more of an enterprise model in terms of administration and security.

So for everybody listening who maybe hasn’t selected an enterprise AI platform, what do you two recommend as maybe indicators that it’s time or maybe even things that they need to think about before they make that decision?

You know, one of the things for me was just I’ll use the word resistance, and I don’t necessarily mean active resistance.

I simply mean despite encouragement, despite promoting it, despite really pushing people to think about how to reinvent themselves.

If people feel resistant, it may be that they’re stuck, right?

Sometimes, it’s not a lack of desire to get on board.

It’s not knowing how to get on board.

And one of the things that Mohan was just talking about that I think is clear, it’s very, very possible to have the security around data that you need without an enterprise subscription, right?

You can buy a personal subscription and upgrade and set settings a certain way and protect yourself.

But I found that there are a lot of people that are still just worried about that and didn’t trust themselves to do the homework and sign up for the right thing and those types of things.

But as soon as we gave them a tool and said, here you go, we’ve signed up.

This one’s good to use and it’s not going to train on your data.

All of a sudden, the people that I had felt were resistant are coming to me saying, hey, guess what I did with ChatGPT and talking about it.

So it was that little thing to empower people, but also to show with dollars and action that we mean it.

We’re not just saying it.

And so I would say if you’re sensing resistance, especially amongst good people, that you’re not used to resistance in and isn’t active, blatant resistance, maybe it’s just that.

Yeah, that’s so interesting.

I still feel like across the board, for anybody that I know of that doesn’t have an enterprise application selected or even guidelines on what to use, what I hear is like everybody just has it on their own, even if it’s on their cell phone.

They’re too afraid to use it on their company computer.

They will just then do whatever they’re wanting to do on their cell phones to keep it separated from their work, which is a really kind of.

Well, the data actually plays that out too.

There’s even a lot of research on this and the silent users are a massive minority.

I don’t remember what it is, but it’s in the tens and it’s not single digits of percentage of people.

Yeah.

There is sometimes the fear, there is sometimes you’re doing it, the silent employee, but sometimes also team members need a nudge.

Because they’re busy, so you’re trying to cram in 50 hours of work and 40, so to speak.

When you do that, there is less time to experiment, and when you just open it up this way and say, hey, this is the new business as usual, people will find the time to invest in doing things better.

So it’s just sometimes just about how crunched they are for time to discover new things.

So Mohan, two questions for you specifically.

The first is just from a chief technology officer lens.

What does a smart rollout actually look like for a platform like this?

You know, I always start off with saying that I hope every organization has a set of business objectives, right?

Whether you use OKRs or whatever methodology, right?

So it’s really important to roll everything out in the context of OKRs.

Otherwise, then what are you doing this with, right?

So it has to be in pursuit of certain goals, business objectives.

Beyond that, I think, you know, if you’re a smaller company, you know, there are enough controls within these tool sets for you to just make sure you take the time, you set the right administrative defaults for everybody in the company.

If you’re a bigger company, you’ll have to set up more governance and training and those sorts of things.

It’s a little more complex rollout, but you can also do it team by team by team.

Then there is the aspect of infrastructure and integrations.

One of the hardest things with these tools is employees level up very quickly, but it is hard to do operations with these because it invariably requires integrations, meaning this system has to push this system through the GPT to do something else.

That is harder.

There is a need for, I’ll call it, that infrastructure integrations, being thoughtful about it, having a team or at least one person who can enable that sort of thing is important, and just kind of monitoring how people are using it.

Are they using it, not using it, understanding where a nudge is needed, or people may be using it great.

So just kind of monitoring it.

And finally, I’d say for leaders, Courtney, that they need to lead by example.

You should be saying that this is a tool.

It’s not replacing what I do, but it’s enhancing how I think and what I do.

And leading by example is super important.

Okay, last question.

For companies listening that are on the fence, what would you say is the biggest upside that they’re missing out on?

Because I think there’s some people that are listening that’s like, well, if my team’s doing it on their phones, they’re still getting the leverage.

But this probably, what are the things you’re missing out on?

I think context matters a lot.

And what I mean by that is, you know, many people have probably seen some of the more recent features and integrations, what OpenAI is calling connectors, out to the ecosystem.

When we signed up to upgrade into the enterprise platform, we actually were given the ability to connect to our Google Drive and capture all of the enterprise information that we already have stored in a secure, private way, so it’s not training models, it’s secure.

But now, it’s almost a knowledge management system, right?

You can go query, and that type of upgrade, that type of enterprise feature that’s being rolled out, I think is phenomenal.

So you may be familiar with some of the personal versions of that, but if you think about knowledge sharing and institutionalizing some of that, and how we can create these enterprise connectors, I think that’s a really compelling add-on that I would say comes with moving from the personal versions to the business versions.

That’s right.

I’ll give a low-tech example.

One of the things that I’ve noticed since we rolled it out is because we are in different places and we work through Slack and email and so forth.

I’ve seen that the writing has become much clearer.

Communications is one of the hardest thing in building a business, right?

And I’ve noticed that communication has gone better because probably what people are doing is they’re putting their thoughts in, their talking points, and just getting it to very succinct description of what they’re trying to say before posting it or emailing it.

And I’d say that’s a low-tech example, but a profound example of how we can make a company better.

You know, one other that I will add to the list is when you’re all using your individual accounts, you have things like custom GPTs that people are building.

And eventually, you move over to the…

The longer you wait to move over to the enterprise, the more knowledge base that you’re actually going to lose by not having already, that may or may not have happened.

Just some people on this call here, a very powerful custom GPT.

Some people.

But I do think the longer you go, you’re going to lose some of those rich tools that you’ve built into whichever application you’re using.

Can I give you a quick tip for those random users who may have experienced some of the memory leakage?

There is a way to be able to have your old ChatGPT instance provide you with an export of data that you can then actually feed to your new instance as just memory capture technique and just kind of have a chat with ChatGPT and just say, hey, I’m moving over my data.

I just wanted to let you know this is what you provided me through that account.

Please reference this whenever possible, however this way.

When you do that, think about the fact you’re going from a personal account.

Maybe you can filter it.

There’s different ways to filter that data.

I’ll use ChatGPT to manipulate that.

But I actually, I had over a year, like months and months and months and months of really depth conversations that I wanted moved over so I didn’t lose it.

And it did a really good job of just kind of chatting with me about that and helping me figure out how to move it.

So quick tip for those that do experience that.

OK, but like you had so much that you literally asked ChatGPT to give you a performance review and…

I did.

They did it very well, you said.

When you say they, what do you mean?

You’re not supposed to agree with that.

True story, David was like, you two will know what it told me I could work on.

And it took us, I don’t think we ever got anywhere close.

It’s just how many flaws I have.

You guys came up with like seven things it should have told me.

I would not recommend doing that with your boss, everyone.

You know, the other thing, though, that I really just to give a real life example, we had a custom David DeWolf GPT and it was super valuable for marketing because we could take in things that we were running to write, drop it in.

It would give us a great first version in David’s voice.

I don’t know why we haven’t talked about that more.

That’s a good use case.

Anyways, we lost that, y’all, and I’m still mourning it, but.

If it was that valuable, I should have told me, we can recreate that.

That’s easy.

We need to.

We do.

Just because I turned off my personal account.

So, we definitely do that.

Okay.

Well, I hope this conversation was interesting for all of you listening.

It’s just a very practical discussion about our decision to move to an enterprise account, kind of the good things that happen with that, maybe some areas to think about as you make that transition.

And hopefully, we’ll help you as you may be moving towards that as well.

David, Mohan, thank you as always.

So much fun.

It’s great.

Hey, guys.

Courtney here, breaking in to share some really exciting news.

As more of our clients have been using Knownwell, we now have the data to show that those clients have seen a 200% increase in healthy accounts.

It’s incredible.

If you’re curious to see what your data would look like turned into intelligence, we would actually love to show it to you.

We’d love for you to actually get your company’s data on the Knownwell platform and see what the health of your clients actually is.

So if you’re interested in doing that, we’d love to talk to you.

You can schedule time with our team at knownwell.com.

Ardy Tripathy is the AI lead at OpsCanvas, a cloud orchestration platform that uses AI to increase visibility, automate workflows, and deliver actionable insights.

This is the second part of his conversation with Pete Buer about how quickly the AI landscape is changing, and how you can decide where AI fits into your overall technology, strategy, and stack.

You referenced the CIDC pipeline, and where applications sit there.

For those listening who might not be familiar with a term, can you contextualize that for us?

Sure, absolutely.

So CICD just stands for continuous integration and continuous deployment.

And so it’s a technical term that basically encompasses within it several processes that happen when software teams develop software and deploy it.

So most software that is written today will be checked into some version control system such as Git.

And CI-CD tools are those that integrate with the Git to be able to automatically deploy, based on the chosen rules, the latest version of the software in the application.

So, yeah, so CI-CD is basically acting like you’re increasing the team’s velocity of shipping new features and also making it more…

You have all the logs present, so you have an audit tree.

And you mentioned in an earlier conversation the notion of zombie resources.

What are zombie resources and where do they show up in this process?

Right.

So zombie resources is one of the things that we identify at OpsCanvas.

And so let me contextualize that with a little bit of the landscape as it currently is.

So Gartner, for example, has estimated that close to 30 percent of cloud resources in companies among different sectors are close to being unused.

So you really have to wonder why is that the case?

I mean, why these companies don’t want to just throw money on the table?

So the answer to that is multifaceted.

So on one hand, with the push to cloud, the cloud hyperscalers, the cloud providers have an incentive to make it easy to use.

So there are tons of managed services.

If you go to the AWS catalog or Google Cloud, and you’ll see so many different managed services, and they can be very easy to start using.

But then that’s good because that enables your developers to think of new features and use the resources and bring and improve your application faster.

But at the same time, it’s easy to scale up, but then when you want to delete stuff, everyone is always a little bit scared.

That’s one of the reasons why you have this middle zombie ground where you have these resources which have been provisioned, but then no one has called them down yet.

By the way, they’re not really doing any good work for you, and they’re just racking up your Cloud.

So these zombie resources are also a security concern, because if you have resources in the Cloud that nobody’s monitoring or nobody is really responsible for, then they are more likely to go out of date, have security issues, and be a point where attackers can access your system.

So these are zombie resources.

And like Jess would say, the best way to avoid zombie resources is to not let them happen in the first place.

And how do we not let that happen in the first place?

Well, OpsCanvas, if you use OpsCanvas in your deployments, then OpsCanvas has a record of all possible deployments that you have done with their appropriate versioning.

And it also knows all the resources that were used in each of those deployments.

So, if you want to scan for zombies, OpsCanvas will go and scan your cloud for you.

It will make a list of all the resources that are there.

And it already knows the resources that are associated with the things that you deployed with OpsCanvas.

So, it will take the difference.

And that difference is our first stab at what zombie resources are.

Now, some of those resources may be legitimate, like you really need them, but you did not use OpsCanvas to deploy them.

And so in that case, you can ignore those.

But nevertheless, it gives you a good first stab of, you know, bringing your Cloud Bill in control.

And if again, back to the audience for the program, if I’m CEO running a business, what should I be worried about and what should I be doing?

What’s your advice to me?

With regards to AI or Cloud?

Well, I was actually thinking, let’s start with zombie resources and then let’s get back then to the top of the order and just see in general what your advice is for leaders.

Sure, sure, sure.

No, no, that’s great.

So I think Jason, who is our CTO and co-founder, he likes to say this simple for W question about your Cloud resources.

It’s basically what, why, who and when.

So basically, if you can answer those four W questions for all of your Cloud resources, you’re in good shape.

The sad answer is most of us, most companies are not.

And devoting precious manpower time to figuring out answers to those questions, it can be a time sink.

So I think there are various tools, like you rightly pointed out, like even beyond AI, like there are various tools in the market that can help you to some degree.

But at the same time, with every new tool, there comes like newer way, newer, you know, new learnings, new trainings that need to happen to be able to use it well.

And who knows, maybe that tool doesn’t work for your case, right?

So we really think visibility is important.

So visibility is answering those four Ws.

And so anything that you can think about if you can ask your teams to make sure that visibility is first priority when they’re even thinking of features, it cannot be after the fact.

Okay, cool.

Then let’s helicopter back up to this top level from a strategy perspective.

We’ve covered how to think about use cases around features and the role that AI can play in getting them up and running.

If I’m a CEO listening, what’s your advice to me for how to think about strategic approaches to getting development deployment management right?

Right.

Strategically and with respect to AI in particular, so with respect to zombies, it’s a more simpler answer.

It’s about cloud visibility, ensure your visibility.

With respect to AI, AI is, like I said, getting better every year or even faster.

Right now, we have inklings off, but we do not yet know the full picture of how AI will be useful in the future.

So I guess one practical takeaway is to encourage your employees to use it within your cost limits and privacy guardrails.

To freely share the wins and describe them internally, to encourage experimentation.

The reason for that is, like I said, AI is an enthusiastic intern, right?

And the best way to get work out of interns is not by mandating that every employee gets two interns.

Now there’s more stuff for every employee to do, but rather just say that, hey, here is a pool of interns, feel free to take them and work with them as you want.

And anyone and everyone is encouraged.

And if the interns do something fantastic, by the way, well, that’s great.

That’s a learning for all the others in the company.

And coming back to that cost point about these models, the costs decreasing.

And so, you know, those internal wins might inspire some customer feature, customer-facing feature, and that customer-facing feature will be cost-effective in the near future because the costs of these models are constantly decreasing.

So I think in that sort of landscape, I feel experimentation is the way to go.

One, when we don’t know what it’s capable of fully well, and we don’t know how much it’s going to cost.

We only know it’s going to reduce.

And is there a right way to set up experimentation?

Like, presumably, you don’t just want thousands of flowers blooming, like there should be some parameters.

How do you think about that?

Yeah, yeah, yeah, yeah.

So the one way is to obviously ensure that you have privacy guidelines.

So if you have proprietary information, please do not use a public-facing, for example, ChatGPT.

That data, you need to be sure about what data access policies are being followed by the model providers.

Maybe for experimentation, you could just use one of the internal models if you have the capability to set one up.

Otherwise, all the companies give you subscription models where they explicitly say that they’re not using your data for training.

Once you have that guardrail in place, the other one is to ensure that the features that are the experiments that are happening are somehow aligned with your product strategy.

That really boils down to the communication from managers, program managers, directors, and so on, all the way to the employees who are actually hands-on with these AI tools.

If that synergy is present, and again, this is not a new problem, this has always been the case about alignment in product initiatives.

But if that synergy is in place, then it is expected to lead to good outcomes.

Another way to have a guardrail, so for example, personally, I work in the engineering team.

We have a Slack channel in which we post about how we are using AI.

We are constantly using AI in our own software development.

And so we just keep posting our experiences there.

And that is one way of communication within the engineering team.

Yeah, Ardy, I noticed that we’re at time, so I’m going to stop here and say thank you very much.

I’m sure everyone who’s listening is going to suit up tonight and go zombie hunting, thanks to the guidance that you’ve given to the world.

Sure, I’m happy to give that.

All the best.

Thanks very much.

Thank you, Pete.

Thanks as always for listening and watching.

Don’t forget to give us a rating on your podcast player of choice, and we’d really appreciate it if you can leave a review.

It’s one of the best ways to help other people find this show.

At the end of every episode, we like to ask one of our AI friends to weigh in on the topic at hand.

Hey, ChatGPT, only fitting to bring you back in for this episode.

Today, we’re talking about what leaders can learn from just 30 days of enterprise-wide AI deployment.

So, what do you think?

Turns out, 30 days with AI in the enterprise is like tossing a robot into the deep end.

Some flail, some swim, and a few start teaching swimming lessons.

Leaders walk away with a crash course in change management, tech reality checks, and unexpected human drama.

And now, you’re in the know.

Thanks as always for listening.

We’ll see you next week with more AI applications, discussions, and experts.