JP Beaudry

Bearing the cross of LLM costs

These days, the only thing worse than failing to leverage generative AI is to succeed at it. Soon after the celebration will come the curse of success: new APIs to maintain and costs to manage. The best protection is hands-down a good set of evaluations. But that could be easier said than done because organization momentum may take you elsewhere. Here’s what and why I’m doing about it. See if that resonates and let me know if I’m missing something obvious!

Two irresistible forces make us all want to upgrade our LLMs. The first, is the great token depreciation we’ve been experiencing and which is likely to continue. Some claim that improvements in the performance of newer small and medium models paired with their lower costs have reduced the price of a unit of intelligence by a factor 1,000x. Maybe. What is undeniable is that, at the time of this writing, the cost of gpt-4o-mini is 66.6x cheaper than not-so-old gpt4-turbo. That’s devilishly difficult to pass up if your constitution allows it!

The second force that’ll bend your knees is when a maker calls one of its models to the great yonder (e.g. PaLM2/bison). Like it or not, LLM API upgrades are as inevitable as death and taxes.

This is where evaluations come in. Or at least, that’s when their importance dawned on me. See, when teams first experiment with generative AI, they are just trying to make it work. They are looking to “build the right thing”. Final judgment, of course,  is from users via either qualitative feedback or measured impact. But when it’s time to upgrade the guts of a product or process already in production with a new LLM, you may be less excited to set up another beta group or run an A/B test. Certainly, I’d rather avoid the time and complexity where possible. If only the thing was “built right” and that a label swap was all I needed. In a good set of evaluations lies the salvation.

In places where LLMs are paired with the other two members of the generative AI holy trinity, prompts and data (input and output), you implicitly have the ability to compare the performance of the new LLM to the old one and, ergo, your eval.

Where that does not exist, one must become the creator. That’s what we’re doing in places like content destined for 3rd party platforms with high-consequence policies or chatbots designed to support precise personas or… The list goes on. Yes, there are tons of awesome and thoughtful public eval sets but they weren’t created for your use case. Some of them may correlate with your use case, but that, too, will have to be tested and discovered.

I wish I had asked the teams sooner to incorporate the creation of an evaluation in their capability designs. I believe that a good eval should now join the set of other things that satisfy the production ilities. Until we atone for this omission, the seductive new batch of models will remain just out of grasp. Don’t say I didn’t tell you so.

How we got started on generative AI

Per a16z, 2024 is the year when enterprises are getting serious about putting generative AI in production. And as I learned a few months ago via an MIT Sloan publication, it’ll be the majority of enterprises. Indeed, according to their published survey results, only about 5% of companies had any sort of GenAI in production in 2023. That figure was surprisingly low to me, given how much the world talks about LLMs. Nonetheless, it derives that many organizations are just now getting started on their AI transformation. As I did years ago about Business Agility, I want to share a few experiential learnings in case they help others. Also, the sooner I create a contemporary body of written work, the sooner I can train an AI agent to write in my stead.

Preamble

For my/our work experience to make sense, I believe some background is needed. If you disagree, feel free to skip over (link) this rather lengthy preamble.

Another potential bone of contention is my credibility on the subject. That’s in the eye of the beholder, I suppose. So here’s some context so you can assess. Today at JustAnswer, our GenAI-powered chatbots help our customers find professional services experts – the living, breathing kind – via 7M interactions per month. Moreover, we have deployed GenAI just about everywhere in our operations, from online ad management to the aforementioned chatbot infrastructure to a more personalized customer experience. Finally, employees across all departments use AI creatively, we leverage LLMs from all the big vendors, and our corporate strategy has had an AI pillar for over 15 months.

As far as my credentials are concerned, it’s admittedly less quantitative. However, I am the company CTO and the executive sponsor for a group of machine learning specialists that drive broad education for all employees, joint experimentation with all functions, and purpose-built systems and services for others to use. So you know that I’m surrounded by early adopters, that I’ve had a first-row seat to the process, and even, perhaps, a little influence.

Before I describe the steps we took, it might be useful to understand JustAnswer’s impetus for action when the power of LLMs became obvious in late 2022 because your situation might call for a different approach.

By now, the market consensus is that some companies will ride the GenAI wave and benefit from it while others will have that wave crash upon them to their chagrin. If there are companies altogether outside the path of the wave, I don’t know of them. But we can agree that not all organizations are strategically equidistant from it (again, context matters), so the urgency for action may differ.

JustAnswer is pretty close to that wave. See, the company seeks to democratize access to professional services by enabling consumers to have their issues resolved by qualified pros across a wide range of domains like medical, auto mechanic, veterinary, antique appraisal, legal, home improvement, legal, and so forth. JustAnswer broadens that access via online matchmaking and lowers the cost by primarily offering a lightweight chat-based interaction.

Questions. Answers. Chat interface. See where I’m going?

ChatGPT coming online was a very loud confirmation of what our data scientists had been saying: AI is getting pretty good at answering written questions. Given the nature of our product, should we not be paying attention? They had been poking around the OpenAI Playground since inception but hadn’t yet been able to convince the managerial class I represent to dial up our investments in AI. But from that point on, it wasn’t a matter of if we were going to embrace LLMs, but rather how.

Enough with potatoes, here’s the meat

When ChatGPT said “Hello world”, we had low double-digit employees sufficiently familiar with generative AI to understand its potential and fewer with any hands-on experience. I cannot claim to have been part of that group. Rather, at the time, I was with the majority of our 1,000 employees. So if we were going to take this seriously, we were going to have to grow our number of employees skilled in AI by two orders of magnitude, starting with yours truly.

The first thing we did was an internal education campaign on what GenAI is and why it matters. Thought leaders and titled leaders started posting regularly on our internal channels. We wrote about what GenAI is, what it can do, and how it could apply to us. Our goal was to create interest. We then created a few dedicated internal channels where technical and non-technical communities could go deeper into their area of interest. This was important because a publication bar set where a post must be of interest to all employees would be discouraging to authors. A bit like I’m feeling now writing this.

After the “tell”, came the “show”. Our most advanced users hosted demonstrations. We started with the basics, such as how to use ChatGPT, what prompt engineering is, what sort of tasks LLMs can readily do, etc. JustAnswer’s employees are a smart and curious bunch, so these demos triggered demand for more hands-on workshops, which our early experts – many in our ML group – so adroitly setup. We owe these folks a debt of gratitude for bootstrapping our GenAI adoption.

By early 2023, the potential of the LLM wave was clear enough that we declared learning to surf it a corporate priority. That declaration trickled down through the management operating system: goals and objectives, investment choices, rewards and recognition, etc. I believe this executive support was essential in overcoming our internal inertia. After all, our leaders and teams were already busy working on plans they liked and which we’d previously approved. Also, it was now clear that AI was going to be more transformative than the run-of-the-mill adoption of a new tool or business process. So we’d be faced with our own internal customer adoption curve, like any transformation. As I wrote years ago (ref), I believe that real transformations require exec support. Luckily, my colleagues on the exec team saw the same need so we were able to move swiftly.

Of particular importance was the decision we made to dedicate an agile team to creating a generative AI “paved path” for others to use. We did not mandate the usage of this emerging corporate capability, but we estimated that while a proof-of-concept is trivial for someone to create with, say, the OpenAI API, production systems are much more complex, what with all the ilities (ref). I may go into detail another time, but in short, we created an LLM proxy service anyone could use. That proxy service became a major enabler. I don’t always make great decisions, but this one has paid off in spades.

Just as important and not much less time-consuming was sorting out the Legal governance for the utilization of LLMs. At the time, it wasn’t immediately clear whether data sent to the big LLM vendors would be used to train future models. Some companies chose to to be cautious as a result (ref Samsung). Being in very close proximity to the wave, we decided to paddle hard instead so we tasked our legal team and ML leaders to generate guidance for the company. I wish I had been more closely involved earlier on because this turned out to be harder than I anticipated. In hindsight, I should have known that two tribes with very different skillset collaborating to clarify an emerging domain was going to require a little get-to-know-you.

Today, LLM API providers have much clearer terms of service and data agreements. Back then, we had to spelunk into each website, proactively request said ToS, and consult with external counsel to pressure-test our conclusions. R&D teams spent quite a bit of time articulating our use cases in terms that made sense to the legal team so they could provide nuanced risk assessments. Indeed, there are different concerns between a marketer creating online content, an engineering refactoring code, and a chatbot asking customers intake questions. I’m not sure why I had the expectation that I had, but it utlimately took several weeks for us to have an articulation that was both agreed by legal and understandable by employees.

So, short story long, by mid-2023, we had LLMs used by employees to refine their craft and deployed in production in a customer-facing manner to great effect. We were now beginner surfers. End-to-end, or rather, from seeing the wave to standing up on the board in public, it took about six months.

Innovation culture and hackathons

This is squarely a humblebrag, but I feel compelled to give a shout-out to the JustAnswer R&D Innovators.

So Proud CTO Alert (📢) out of the way, I want to shine a big spotlight on the JA R&D team for stunningly inventive demos during our 42nd quarterly hackathon. That’s right, 42! It’s a true testament to the company’s dedication to exploration and professional growth, and, according to HHGTTG (ref), perhaps even the meaning of life.

This quarter, we devoted our creative energy to going big with Large Language Models (LLMs). In an era where AI offers immense opportunities and complex challenges, focusing on LLMs was natural and timely. The results? Absolutely inspiring – 79 of our talented team members came together to create 37 mind-blowing demos. Seeing so many of these ideas getting ready to make their mark in our day-to-day operations will brighten anyone’s day.

Having joined JustAnswer a little over a year ago, I’ve been consistently wowed by our agile, hypothesis-driven approach to product development, what with its rigorous multi-variate testing and rapid iterations. But you know what I think the real jewel in our crown is? Our unwavering commitment to learning, brought to life in the company’s lean philosophy and practices like these hackathons.

For employees, a hackathon isn’t just an event – it’s a reflection of a decade-long journey in embracing agile and making it a core part of our DNA. This latest LLM chapter of the journey started with our ML specialists exploring the OpenAI Playground as soon as it launched. Then came the educators, shining a bright light on the crazy potential of this technology via company-wide lunch-and-learn sessions. The innovators followed, crafting prototypes, how-to guides, templates, and documentation to give everyone a running start. Our legal team then chartered the waters to ensure we could navigate safely and wisely. And let’s not forget the early adopters who delivered stunning production results and the decision-makers who bravely prioritized these groundbreaking experiments. It’s been a remarkable team effort all the way.

A huge round of applause to everyone who turned this cutting-edge technology from mere concepts into borderline-magical solutions! Navigating the AI wave requires skill and grace, and you’ve all shown that in spades. My next challenge? Selecting the hackathon winners. Though in the grand scheme, that’s a great problem to have.

Here’s to continued innovation, relentless learning, and more groundbreaking hackathons!

Why Go To Industry Conferences

I like industry conferences.  Recently, I participated in Agile 2016, held this year in Atlanta. As I reflect on this trip, I’m amazed at how much value I got out of it. I count at least five ways this trip was beneficial. This feeling is consistent with other events I’ve participated in, be it Agile Games, Agile New England and others . I’d like to describe why I get so much value out of attending conferences. Perhaps it will make you want to try one, in case you haven’t yet.

Value As Attendee

The most obvious value I get out of conferences is as an attendee. There is typically a large selection of relevant and well-presented topics delivered by passionate people. The presenters are sometimes industry thought leaders. I love to have a chance to experience them first hand. And frequently, I can meet and talk to them. Not only do I walk away with new information and ideas, but also the presenter’s energy gives me a sense of renewal for whatever cause I’m pursuing.

Value As Presenter

When I can, I give back to the community of interest I am part of by sharing my experience. For example, at Agile 2016 I co-presented how we have evolved our internal Agile education to make it more effective for learners. It sure is hard work to prepare a workshop or presentation. And there is no doubt that it can be stressful. But, as the old saying goes: “If you want to master something, teach it”. There is no better way to organize your thoughts and deepen your understanding than to instruct other people. The Agile community is particularly good at providing constructive feedback. I always walk away stronger.

Value Of Networking

It’s so easy to meet other professionals at a conference. Not only do we share an interest in common, but often we’ll be wearing name tags. Getting introduced is thus a breeze. Making connections is great because I frequently stumble on people who have solved problems I’m wrestling with, or who face issues I can help with, or who could eventually be business partners. Think of it like LinkedIn, only much high bandwidth, quicker, and more personal

Value Of Business Development

As I attended a recent conference, I was also wrestling with a business challenge for which I was looking for expert assistance. Because a significant portion of the world’s experts was in attendance, my team and I were able to organize four introductory conversations in the short span of two days. We were quickly able to identify two providers that could help us, and we may soon hire one of them. That same sequence of events would have taken weeks of wall clock time had it been done from the office.

Value Of Fun

At many events, and particularly at Agile 2016, the organizers did a marvelous job teeing up activities that encouraged people to unwind, relax and build relationships. I can be skeptical of those socializing events, but I have to admit that I usually have a great time. And it’s nice to have something fun to do, especially when you are away from home.

Bottom Line

In short, between learning through others, learning by teaching, building new connections, making progress on business issues, and having fun, I get a ton of value from industry conferences. I plan to continue to attend, and I sure hope that I will keep on being invited to present to share my experience. Bottom line: participating in conferences has made me a more valuable change agent. I think you would find the same.

Get More From Your Devil’s Advocate

Across all industries, there is a tremendous desire for more innovation.  Sadly, companies frequently suffer from a cultural habit that stymies creativity: people play the role of Devil’s advocate in a non-constructive, unidimensional fashion. New ideas face a constant barrage of negativity, and promoting them becomes a Herculean task that discourages but the most intrepid innovator. With a bit of training, this well-meaning but drag-inducing scrutiny can be redirected to identify and strengthen the best new ideas. In this article, let’s define the problem and introduce a few solutions.

An Example

I was recently discussing an issue related to our cultural transformation with colleagues.  As I advanced a potential solution, someone barged with “let me play devil’s advocate for a minute”. The person then proceeded to cast F.U.D.  upon my idea. I was instantly annoyed: the person hadn’t taken the time to listen to the whole idea nor, seemingly, to consider its merits. Even ignoring the impact this rejection had on my ego and the potential consequences on my future participation, the person was acting as a net brake on forward progress. The transformation issue was no closer to being resolved after that person’s interjection.

Overcoming Inertia

Unfortunately, some people mistakenly believe that the devil’s advocate’s main responsibility is to oppose (mostly with good intent, more on that in a moment, please hold your nose). Because of that, the role often ends up contributing to maintaining the status quo instead of progress.

The most difficult thing about cultural change management is overcoming inertia, skepticism, and opposition. Therefore, input that solely seeks to oppose or dismantle is not tremendously valuable. The universe is great at disassembling things all by itself. Indeed, the 2nd law of thermodynamic  states that “that there is a natural tendency of any isolated system to degenerate into a more disordered state”.  Mother Nature gives us disorganization for free. She doesn’t need help.

What we need is construction, design, and creativity. These things require action, drive, and energy. And we should expect our Devil’s Advocate to contribute.

Intentional Validation

You can let go of your nose.

To be sure, there are absolutely times when we need to look at reasons why something won’t work. Yes, it is critical to pressure-test the viability of ideas and to identify their riskiest assumptions.  This is best done deliberately.

For example, in the space of entrepreneurial ventures, Jim Mullins provides seven domains through which an idea’s killer assumptions can be identified (see his “New Business Road Test” book . The well-known Lean Startup  is all about reducing risk through experimentation. And in law, preparing someone to testify requires a similarly purposeful preparation and line of questioning.

In all cases, the pressure testing is calculated, focused, and exhaustive. And it happens *after* the value of the idea has been deemed high enough to warrant such an examination. It’s never mere opposition.

OK, I promised a few approaches. Here they are:

Six Thinking Hats

One of my favorite techniques to avoid the opposition trap is the ‘Six Thinking Hat’ technique. The creator, Edward DeBono, suggests that all conversation participants simultaneously look at the issue through a common but rotating lens.

For example, all undisputable facts about the issue are listed when participants figuratively wear the white hat. No room for opinions. After switching to the yellow hat, everyone chimes in with the value and benefits they see in the idea. Eventually, all participants are asked to wear the black hat of judgment, which enables them to rattle off all the pitfalls of the idea. And so on for the other hats. At the end of the process, everyone is assured that all sides of an idea were thoroughly explored.

Design Thinking

The Design Thinking method includes a similar form of lateral thinking . It encourages interlocutors to reply to an idea in the form of: “I like, I like, I wish”. For example:

Jack: “We should hire an Agile coach for every team” 

Jill: “I like how this would accelerate the transformation. I also like how it would make expertise available to all those who want it. I wish we could validate the model before making such a large commitment.”

Participants have to acknowledge the valuable aspects of an idea before highlighting a risk, and then only in the form of a problem to solve. It’s very constructive.

Improvisational Comedy

To enhance collaboration, many entrepreneurs are borrowing a technique from improvisational comedy known as “Yes, and…” Seth Stevenson  describes it as:

When they’re collaborating onstage, improv performers never reject one another’s ideas—they say “yes, and” to accept and build upon each new contribution. “It’s a total philosophy of creativity,” says Mandel. “ ‘Yes, and’ creates, while ‘no’ stops the flow.”

The method strengthens ideas, rewards listening skills and avoids the pitfall of a ‘no’ culture. With many people having a hand in the creation of the idea, the downsides and risks can be explored without fearing to choke the idea to death.

Owning Forward Motion

Each method explored above includes an affirmative responsibility to look at the potential of an idea prior to any pressure testing taking place. This ensures that forward motion is preserved.

Once demotivated by careless rejection, most people cannot muster the activation energy required to carry their ideas forward. It is imperative that we remind each other that we are mini-CEO’s, each tasked with bettering the current state of affairs. The Six Hats, Design Thinking’s “I like, I like, I wish”, and “Yes, and…”  are simple but effective tools with which we can get more from our Devil’s Advocate. Have you experienced others methods to achieve the same goal? Have you discovered immunity to this problem?

Works cited

“Lawyer”, by Rflor, The Noun Project, Web. 20 Jun. 2016.  Modified.

Beware Of Identity Theft During Agile Coaching

All Agile coaches eventually face the stiff headwinds of resistance to change. Counter-intuitively,  coaches themselves can trigger some of that resistance by inadvertently performing wide-scale identity theft. It is something I have done, and you may have too. This article explains what identity theft is, provides some concrete examples, and looks at a few approaches to mitigate the issue.

What It Is

Coaches need to acknowledge that people derive their identity in part from what they do. A quick look at what LinkedIn and Twitter users put in their profile demonstrates the point: author, database administrator, director, entrepreneur, hacker, manager, programmer, statistician, etc. People put a lot of energy into creating their identity. Therefore, any change management initiative that threatens that identity, whether knowingly or not, will be met with resistance.

Concrete Examples

Here are some examples of identities that Agile coaches are susceptible to attack:

Coaches frequently form teams of people and urge them to swarm on issues. This is done for good reasons. Nonetheless, how might it come across to people who have spent the previous ten years becoming experts in a given technique? Take the developer who specializes on the Java stack, who attends advanced training, who eagerly awaits and learns the features of each new release of the language. How does that person view the suggestion that she should learn and fumble with front-end development or database schema optimization? What does it do to her self-image of being an expert?

Most Agile methods do away with big up-front planning for a more just-in-time (JIT) approach, which is very sensible in situations of high uncertainty. However, we shouldn’t be surprised that a JIT approach triggers an adverse reaction on the part of people who are professional planners. You may have met Lucy, the program manager, who is the go-to resource for any questions related to Microsoft Project. How will she feel about the idea of abandoning detailed planning?

Then there is Mark, the software architect, who can produce exquisitely detailed system functional documents for the team to implement. And there is Stephanie, the lead quality assurance engineer, who authors thick test plans for other people to execute. As a coach, you may have inadvertently devalued the activities with which they associated the most. A similar thought process applies to other roles such as the traditional manager, the financial planner, the requirement translating analyst, etc.

Modified Coaching

One way for the coach to avoid the identity theft land mine is to lead with principles and values, not solutions. Let’s hark back to the example where a coach might propose swarming. An alternative approach is to see if the team members prize speed. If they do, might they have ideas for how to increase it? Frequently someone will suggest swarming, but you may be surprised with other creative approaches. The point is that if the idea originates from the team, it will be more readily accepted, and the coach will no longer be the source of friction.

Lightweight Method

There are many Agile methods out there, each with their pros and cons. In environments where identity theft is a concern, a coach would be well advised to consider a method that can be rolled out with minimal disruption.

One method that is careful to reduce disruption is Kanban. Just look at the first step of deploying it: “visualize the current workflow”. No change required to getting started. David J Anderson, chair of LeanKanban, goes a step further and proposes â€œrespecting existing roles, responsibilities and job titles” as a change management principle. In environments where the willingness to change is not a given, this seems like wise advice.

A Little Sensitivity

In summary, it is critical for Agile coaches to recognize that people identify with what they do and that change management efforts sometimes threaten that identity. A thorough Lean/Agile transformation affects many traditional roles and will thus have many opportunities to step on identity-based land mines. Coaches are therefore urged to exercise sensitivity and use a principle-based approach. In environments where the appetite for change is limited, coaches should consider using methods that minimize disruption. Doing so should make the path to enhanced business outcomes smoother. What other tips do you have to ease people into the notion that their role may evolve?


Works cited

“Id Card”, by Creative Stall, The Noun Project, Web. 4 Apr. 2016.  Modified.

Agile Rituals Improved – Preparation

This article is the second of a series of two on boosting the effectiveness of Agile rituals. The first article emphasizes increasing people participation while this article focuses on meeting preparation.

Preparation is important because a lack of it can result in long, boring, clumsy, ineffective rituals. Among other things, I think this is partly why famed coach Allistair Cockburn sometimes provocatively asks the following in his training: “Agile is less efficient than Waterfall because?” In any case, here are some tips to avoid the self-inflicted wound of sloppiness.

Note: As in part one, this article assumes the reader is familiar with Agile, and that participants have recognized that their rituals need help and are willing to try new work methods.

Tip #1 – To speed up Scrum backlog grooming and planning meetings, have the Product Owner send the stories to be covered 48 hours ahead of the meeting, and set the expectation that team members read them. This pre-screening allows you to skip the deadly read-out and enables the discussion to commence immediately. You’d think that a public backlog would make this automatic, but sometimes the organization needs a little help developing the habit.

Tip #2 – Did you ever witness a public argument between some or all of the members of the servant leadership team (architect, manager, PO, and Scrum Master)? Yeah, me too. It is not pretty. It is also hypocritical because those same people then ask the teams to self-organize, which is something the leaders just demonstrated they couldn’t do themselves. To avoid public arguments that confuse team members, servant leaders must meet before all major rituals and sort out any sticking points.  Private debate and public alignment are the order of the day.

Tip #3 – One person should be in the ritual room 10 minutes ahead of the meeting time to get the technology up and running. Also, don’t ever attempt something in front of a large group without a dry run first. Just don’t do it. Murphy’s law won’t allow it.

Tip #4 – Don’t wait for people to stroll in to get the ritual underway. Get started exactly on time. The majority of participants will appreciate it. Any tardiness will fix itself after a few people have to walk into an active session. This is especially important when customers are present.

Tip #5 – Don’t get delayed by laptops being swapped out in the middle of a presentation. Do exchange and consolidate files ahead of time. Better yet, consider not using slides. That’s just one of the many fantastic communication tips from Training From The Back Of The Room.

Tip #6 – If different computers absolutely must be used during a demo, have people pre-login applications like remote desktop to make toggling a breeze.

Tip #7 – It is not always easy for remote participants to determine that they are causing distracting noises. Background noise, keyboard clicks, coughing, chewing and ringing phone can really diminish the quality of the communication between sites. As the number of participants increases so does the difficulty of maintaining a proper mute/unmute state. Consider using someone with admin rights over the communication tool to help mute unwanted sources of noise and preserve people’s dignity.

Tip #8 – Conversely, pay attention to local sources of noise and position microphones carefully. Food with crinkly wrappers, equipment fan, shifting chairs, and fingers tapping tables can be distracting to local participants and prevent remote attendees from hearing. In fact, establish your own feedback loop by having a local leader call in from a nearby room to experience the event first-hand.

Tip #9 – The pressure of organizing an event can sometimes cause key meeting artifacts or important activities to be overlooked. Do use an agenda checklist for all rituals. A public checklist also subtly cross-trains participants to self-serve.

Tip #10 – Don’t trust that yesterday’s prototype will just work today. The larger the team, the less likely this is. Do use lock and key for physical prototypes, and freeze the assets for digital ones. This is especially important when the ritual is costly to reschedule, such as some customer demonstrations.

As a change agent, you should be mindful that the emerging Agile culture is not torpedoed by clumsiness. In most cases, the only thing missing to achieve effectiveness is the drive to have it. That drive is embodied in a professional level of preparation. We covered several actions that can be taken to have shorter yet more effective Agile rituals, and most have very low barriers to execution. How else do you like to see your teams and leaders prepare?

Works cited

“Perfect Score”, by Evan Shuster, The Noun Project, Web. 30 Mar. 2016.  Modified.

Agile Rituals Improved – Participation

Let’s face it; many Agile teams are disappointed by how much time they spend in rituals. But, it doesn’t have to be that way. In fact, it shouldn’t be that way. A team can take many actions to shrink the amount of time it spends in rituals while simultaneously increasing their effectiveness and fostering better engagement. Those actions tend to fall in one of two categories: participation and preparation. In this first of a two articles series, I am sharing a few examples of participation-related practices that have worked for me. A subsequent article will look more closely at preparation.

Note: despite some Scrum references, the tips can be applied to rituals from any Agile method. This article, therefore, assumes a certain level of Agile literacy. It does not describe the basics of the various rituals discussed.

Before changing anything, the first step is to ask the participants if they are satisfied with the current state of affairs. If there is no dissatisfaction, there will be no willingness to experiment with solutions. For the purpose of this article, let’s assume that someone has secured buy-in.

The first of the two principal causes of long, boring, ineffective rituals is the dearth of active participants. Here are some tips in no particular order:

Tip #1 – During grooming, don’t let the Product Owner do all the talking. A few days before the ritual, look for team members willing to learn a few stories/epics and take the first crack at answering questions about those during the ritual. A variety of presenters make for more energizing meetings. And you should walk away with at least one person who has a deep understanding of the topic because this technique is an embodiment of: “If you want to master something, teach it.”

Tip #2 – One of the surest ways to lose meeting participants is to have them wait on someone taking notes. This frequently happens in Scrum grooming and planning because the PO is trying to facilitate the meeting, describe the user stories, take notes, and update the backlog in real-time. To cure this, delegate tasks to the Scrum Master and any chickens present in the meeting.

Tip #3 – Story sizing debates derail many a Scrum planning rituals; is this story 3 points or 5 points? There are endless opinions. What participants frequently forget is that sizing is an exercise in estimation and that all estimates are wrong to some degree. Therefore, only the order of magnitude truly matters. For example, teams using the Fibonacci series don’t have to agonize over whether a story is one size up or one size down. The key is to limit the debate to arguments that are likely to sway the estimate by a lot.

Tip #4 – In every group meeting some conversations devolve into back and forth between two or three participants while everyone else waits. To snuff this, devise a mechanism where the crowd signals its desire for the conversation to stop. For example, people can raise a hand, and when two or more hands are up, the debate has to pause while a check for relevancy is performed. In short, don’t let the audience be victimized; empower it to self-regulate.

As a leader, bad meetings are just too expensive; there is the immediate cost of wasted time and the longer cost of reduced employee engagement. Moreover, group meetings are a golden opportunity to influence the organization’s culture. Servant leaders have a responsibility to work toward ameliorating the environment (see the article on the role of the manager). Some of the tips above may be useful in that pursuit. What other tips do you have?

Works cited

“Citizenship”, by Alberto Miranda, The Noun Project, Web. 30 Mar. 2016.  Modified.

Definition Of Done Mindset

A fear often associated with self-organizing teams is the potential loss of focus on quality. Indeed, how does an organization ensure it produces a satisfactory level of quality when that responsibility is distributed among teams rather than centralized? Many organizations solve this issue by crafting and abiding by a Definition Of Done, or DoD for short.  A DoD is to product developers (analysts, designers, marketers, software programmers, testers, etc.) what the National Electrical Code (NEC) is to electricians. Unlike the NEC, however, there is usually not an independent association telling an organization what the DoD should contain. Each organization has to figure out craft its own DoD. In this article, we explore the mindset that should be adopted when crafting a DoD.

Before diving in, let’s remember that a Definition of Done is nothing more than a list of feedback loops a team deems necessary to address in its quest for quality. For example, teams that operate in a software product development context could get inspiration from this DoD starter kit.

Realistic or Idealistic?

It can be tricky to establish a Definition of Done while product development processes are still evolving. Should a team adopt a DoD that can be met today, even if it compromises on quality a little, or should it adopt a stricter, more thorough DoD that the current reality prevents it from meeting? I have a strong personal preference for the more stringent DoD, the one that aims for perfection.

Presuppositions

At this point, I must make a couple of assumptions explicit. First, I’m assuming that an organization embarking on a Lean Startup or Agile transformation is an organization motivated to change. Presumably, its leaders are dissatisfied with current results and have the willpower to plow through the inevitable obstacles (see Executive Support article for other willpower manifestations). I am thus assuming commitment. We will see in a minute why this is critical.

Second, the stakeholders must recognize that a cure for the current organizational ailments cannot be found in the mere mechanical adoption of an Agile method, be it Kanban, Scrum, XP, or any other. There must be a realization that the method’s chief promise, especially when it comes to quality, is to highlight the problems from which the organization already suffers. Only a change in mindset will allow the organization to solve those issues, not some Agile “pixie dust.” I am thus also assuming intellectual honesty.


Problem Solving

A committed and candid organization wants to be aware of its biggest issues as soon as possible so that it can get on with the arduous task of solving them. Through the prioritization and resolution of its most significant impediments, the organization knows it will get stronger and better able to meet its goals (see article on Agile transformation motivations). Such an organization would, therefore, craft a DoD based on what it needs, not what it can currently achieve. Put another way, theDoD will be measured against perfection, not benchmarked against current capabilities.

Pragmatism

Before going further, I must define what I meant by perfect quality. I am not talking about a blind aggregation of all techniques known to mankind that could possibly drive quality up no matter the cost. Here, perfection aligns with customer expectations and the organization’s economic context. If a given quality dimension isn’t important to customers, the DoD should relax or omit criteria along that dimension. Often, organizations will describe the quality context in a QA Vision or a Quality Management Strategy.

At first, a perfect DoD may be difficult to meet due to existing impediments. It’s even likely that some elements of the DoD cannot be met at all if frequent releases of valuable increments are to continue. That’s OK. Resolving those impediments is presumably valuable (otherwise they wouldn’t be impediments). That value should be estimated, perhaps with a tool like Cost of Delay, and the resolution of the impediments prioritized against other initiatives. In the interim, unmet “Done” criteria should be treated as explicit, temporary exceptions. It is important that the capability gaps remain visible. Team members have frequently surprised me with creative and elegantly simple solutions. Conversely, if a difficult-to-meet criterion is removed from the DoD, a team will soon forget that this problem remains to be solved. Out of sight, out of mind, and all that jazz

In summary, my recommendation for an organization establishing its Definition of Done is to ignore current impediments, however intractable they might seem, and write its ideal DoD. I believe this promotes a level of transparency and candor that cannot be attained when starting with a merely achievable DoD. In other words, I’m asking that teams channel famed American football coach Vince Lombardi who said: “Perfection is not attainable, but if we chase perfection we can catch excellence.”

Works cited

“Checklist”, by Yamini Ahluwalia, The Noun Project, Web. 28 Feb. 2016.  Modified.

What Executive Support of Agile Means

Executive support is often touted as one of the most important, if not the most important pre-condition for a successful organizational transformation, be it Agile or otherwise. I agree. However, the nature of that support is not frequently defined. Worse still, I believe it is for most part misunderstood, especially by the executives themselves. In this post I cover some aspects of that support as I’ve encountered it in the wild.

What It’s Not

Let’s start with one thing executive support is not: a passive state. Support does not mean merely tolerating Agile (or Lean, or whatever; I’m not a stickler) in the organization, or just giving the green light for other people to change. It is not something to be exercised from a distance through intermediaries. I understand this may come to the great chagrin of some executives who were content to let things happen down there, way below them.

No, executive support is an activity, a role, a contact sport. The main reason is quite simple: organizations have a tendency to mirror their leaders. Thus, by definition, if an executive does not adopt Agile-compatible behaviors, neither will the organization.

Values

For example, it is critical for the executive to decree new, Agile-compatible values for the organization, and articulate the reasons why they are being adopted. If a senior leader cannot explain the sought-after benefits of the transformation, it will be difficult for employees to see the transformation as anything other than a passing fad. One caution: the leader should stop shy of instituting a specific Agile method. This is for the rest of the organization to figure out. Not only is the executive not in the best position to know which method is called for, but also the organization would then be even more susceptible to blind mechanical adoption (i.e. Cargo Cult).

Active Support

Indeed, the identification and decision about which specific Agile method the teams will use should be pushed down to the lowest possible level. Ideally, it’s a determination best left to the teams themselves, after they’ve received a modicum of education on Agile values, principles, and methods. By pushing that decision down, the executive shows support by starting to demonstrate attributes of servant leadership, a mindset that will have to be internalized by many people in the organization if it is to maximize its potential. (We touched on servant leadership in this article.)

Another critical area where an executive demonstrates support for the transformation is on new and evolving job roles. For example, the Scrum method calls for new titles, like Product Owner and Scrum Master. Those roles will have to be defined in their scope and career progression, and proven “safe” for the first few employees who raise their hands for these roles. Correspondingly, some historical roles will evolve, sometimes a lot. For example, the traditional project manager who does a lot of intermediation may now be deemed counterproductive. Similarly the micro-manager and the lone wolf engineer may no longer be viewed as assets. Indeed, if an executive tolerates historical behaviors that are now harmful to the new culture, employees will deem this a lack of commitment. They will note the irony when their commitment is then sought. Moreover, the executive has to make sure that the good people, whom the rest of the organization knows better than the exec, have their fears and questions answered through this time of change. It can be a difficult balancing act.

Resolve

Finally, an executive must maintain a firm hand on the culture change tiller when the waters inevitably get rough. I strongly agree with the old saying that the main promise of Agile is to shine a light on the problems the organization already suffers from. Some of those challenges will be big, and this may be discouraging to the people of the organization. At that point, there is frequently a tendency to ignore the inconvenient  Agile principles and to implement workarounds instead of confronting the difficulties head-on. Executive support requires that one maintain the pressure to change.

I think there can be little doubt that executive support is much more than a mere agreement to go ahead with the transformation. It involves signaling and modeling the change from the front, while ensuring that the others actually follow. It is a key transformational success factor. Besides the aforementioned examples, what other major actions should an executive take?

Works cited

Support”, by Gregor ČreĹĄnar, The Noun Project, Web. 30 Jan. 2016.  Modified.