Disclaimer: The views expressed in this blog post are fully my own.

AI Code Assistants are all the hype at the moment – and that includes large enterprises, which are quickly adopting and rolling out these new breed of tools to their hundreds, if not thousands, of developers. It’s clear why: real-world companies are already seizing the massive opportunity to innovate and ship much quicker to market (reporting up to 50% increase in PRs and builds), while developers enjoy a more fulfilling job.

Since GitHub Copilot was shipped to market, I’ve overseen and guided rollouts anywhere from 200 GitHub Copilot seats in a single country up to 15,000 spread across the globe. However, in rolling out AI Code Assistants in the enterprise, I’ve often noticed how initial enthusiasm among engineering leads and developers can quickly give way to some amount of disillusionment. This often stems from:

Developers not rushing to grab licenses and make use of the tool as expected.
Code generation that does not fit the developers’ requirements.
A perception of unfulfilled business value promise.

These three issues, in my opinion, arise from a lack of proper rollout planning - and I consistently turned them around by doing just that: planning them properly. In this blog post, I will share with you key lessons I’ve learned about successful AI Code Assistant rollouts – so that you don’t lose time (and money) in unlocking the next level of developer productivity. This is based on my real, ground-level work with both mid-sized and global companies. I’ll use examples based on GitHub Copilot Business.

What a Developer Needs to Know

For as much as AI Code Assistants may sometimes look like magical tools, their successful adoption by engineering teams is far from magical. Developers need to be enabled on 3 key topics for productivity gains to quickly start showing up:

The obvious one: they’ve been granted a license!
Setting up the AI Code Assistant in their specific environment
Using their new shiny tool effectively

This may seem obvious to you, but in my experience, they’re the first major trap enterprises fall into - which typically results in early frustration for all parties and important delays realizing business value.

Obtaining and keeping a license

The fact that a developer has been granted a license should be clearly and promptly communicated. What’s more, if there are any rules by which you want to “gate” a developer obtaining (or losing) their license, or any internal “license request” process, these should be documented and communicated as well. This should be plugged into your communication plan - I’ll cover that in the next section.

The email I got when I was assigned a GitHub Copilot license. Helpful and to the point. Note that XCode is also supported today.

If you use GitHub Copilot, an email is automatically sent to each user that you assign a license to. This email contains, in addition, links to instructions on how to set it up for each supported IDE, which takes us to the next topic.

Setting up the AI Code Assistant

Developers need to be shown how exactly to set up the tooling in their specific environment. Some may be using VSCode, some others Eclipse, and some Jupyter Notebooks. Those working in the office may be behind a corporate proxy server, and those at home connecting through a VPN. Ultimately, they all need to:

Install the AI Code Assistant on their IDE,
Configure any special networking settings and
Get authenticated with their licensed account.

Many will figure all this out alone, but many will not.

Using the AI Code Assistant effectively

Finally, the moment of truth: writing code with AI assistance. In some situations and for some developers, magic will start happening pretty quickly and almost seamlessly. In my experience, this will typically be the case for:

Developers bootstrapping new projects
Developers working with a language and a stack that’s wildly popular in Open Source
Those who used the same tool on their own and/or are plugged into the AI revolution

In an enterprise, however, the most common scenarios are nothing like the above:

Developers will mostly work on large, legacy projects
Developers may often work with unpopular languages, with sometimes closed source as well as in-house frameworks/libraries
It will probably be the first time they code accompanied by an AI

So for 99% of enterprises, a developer enablement plan needs to be put in place. This does not need to consist of countless hours of in-person training nor endless virtual demos. At its core, the plan can be split in 4 tracks to cover the basics - and extend as needed once results prove you need to go farther:

Basic Functionality: how the assistant can be used to generate code.
The art of Prompt Crafting: as LLMs powering these tools are probabilistic, developers need to know how to optimize their prompts and the context to get the output they seek.
Task-specific functionality: best practices to get the most out of the assistant for specific tasks such as writing tests, debugging code or finding your way out in a legacy codebase.
Domain-specific usage: deeper dive to learn how to use it for specialized developer domains such as Data Science, Mobile Apps or Deploying to the cloud.

Components of an enablement plan on AI Code Assistants for developers

The first three tracks are applicable to every enterprise rollout, while domain-specific modules will depend on the characteristics of the development work taking place at your enterprise.

What an Engineering Lead Needs to Put in Place

As an Engineering Lead, you want your developers to thrive and build great solid stuff that ships fast. That’s why you advocated for rolling out an AI Code Assistant. And as we just learned, you’ll need to make sure your developers know how to access and use it. Your tasks can be summarised as:

Putting together a Metrics Framework
Setting up enterprise-level configuration
Assigning and managing licenses
Crafting and deploying a developer enablement plan
Communicate, communicate, communicate

Let’s have a quick look at them.

Putting together a Metrics Framework

As an Engineering Lead you have to justify the investment on an AI Code Assistant. You probably did this based on the promise of increased developer productivity, and now it’s time to deliver and show results in order to keep and even grow that investment. You will do this by setting up a Metrics Framework consisting of:

Your Goals, linked to adopting the AI Code Assistant, for example:
- 50% more PRs
- 35% suggested code acceptance rate
- 90% of developers expressing more fulfillment at work

Industry benchmarks (when available) to reality-check your goals, example:

Case studies from similar actors in your industry (e.g. Accenture case & video)
Industry averages if your vendor can provide them

Metrics to follow on those goals

Quantitative: automate as much as possible using APIs
Qualitative: use regular developer surveys
Pick up a reporting cadence

A GitHub Copilot quantitative metrics dashboard from the open source project microsoft/copilot-metrics-dashboard

You should get all stakeholders aligned on the above framework and report at a regular cadence. The results you obtain will allow you to iterate and adjust your overall rollout strategy if needed.

Setting up enterprise-level configuration

Most AI Code Assistants will have an initial enterprise-wide setup or configuration to make. On GitHub Copilot, for example, you need to set up policies that control which specific functionalities are made available and which organizations should get access to the tool.

Assigning and managing licenses

Here is where you chose your license rollout model: do you want to go “big-bang” and get everyone a license immediately, or do you want to start smaller and increment gradually? How will you manage unused licenses? If you are in a large enterprise, will you bill each department internally or pay centrally for all licenses?

If you put together a strong rollout plan as I’m suggesting in this blog post, and you have the resources to make it happen (especially when it comes to enabling developers), there is no reason not to go “big-bang”. If you are limited on resources, you may chose to enable X number of developers at a time in waves - but be careful on enablement overhead costs.

On GitHub you have tools to manage GitHub Copilot licenses programmatically as well as to support internal billing if needed.

Crafting and deploying a developer enablement plan

See what we discussed earlier and put all sessions in a calendar based on your adoption goals. Make sure you record them and put them into a well-advertised self-service platform.

Communicate, communicate, communicate

It is difficult to overstate how important this one is, especially within very large and/or geographically distributed developer organizations.

Your communications plan shouldn’t be an afterthought: it has to be defined prior to license activation, and it should consist of at least the following campaigns:

Announcing the license activation date and how to get set up
On-demand enablement: useful resources
Enablement sessions: content, dates, signup if any
Surveys and results

For this to be effective, you should make sure you have the reach needed to get to all your developer base - and if you don’t, you’ll need to secure it by working together with other leaders within your enterprise.

Iterate and adjust

Our industry moves extremely fast and the AI field even more. Whatever assistant you choose, it is very likely it will get new functionality added and modified very frequently. You will need to iterate your rollout plan not just considering whether you’re far from your stated goals, but also the evolution of the AI Code Assistant itself!

Summary

In short, enterprises are adopting AI Code Assistants to innovate and ship products faster, with up to 50% increase in PRs and builds. If you want to reap the benefits quickly without losing time and money, you need to get the following right:

Developer Enablement: Key areas include obtaining a license, setting up the tool in various environments, and learning how to best use the tool. This is crucial to avoid frustration and slow uptake.
Engineering Lead Tasks: Includes enterprise-level configuration, license management, rolling out developer enablement plans, setting up a metrics framework, and continuous communication.
Metrics Framework: Set goals, use industry benchmarks to keep real, and follow both quantitative and qualitative metrics to iterate, adjust and justify/grow the investment.
Communication: Essential for successful rollout, including announcing license activation, providing resources, and conducting surveys.

I hope this helps!

Thanks for reading.

How to adopt AI Code Assistants at Enterprise Scale