DevLearn has always been my favorite L&D, elearning-nerd out conference. It’s where I first hit the stage back in 2009 with my Kineo colleague with Stephen Walsh and then most every year until about 2019. And then we had a pandemic and I sat out a few years. 2023 was my first DevLearn since 2019 and it was really great to be back!
- Kineo was a proud sponsor of DevLearn this year and it was super cool seeing our logo at the registration stage and on the big screen.
- Amazing to be on the ground with three of my Kineo colleagues: Joe Replin (Technical Consultant coming to Vegas from Chicagoland), Rodrigo Bolanos (Head of Kineo LATAM from Argentina), and Vicky Bartolacci (Kineo Managing Director all the way from Sydney Australia).
- So great to connect with so many amazing people – old friends and new connections made.
- Data and Analytics are top of mind for people and extremely low on the L&D team’s list of skills. Orgs and individuals are eager to build this capability.
- AI was the darling – and it’s coming soon to a workflow near you if it hasn’t already. AI is transforming how we work and what we’ll be delivering to learners. Game changer. People are feeling a lot of uncertainty and fear and…yes, excitement. Hold on to your hats, the ride is just getting started.
- Ooh – and I signed a whole bunch of copies of The Accidental Instructional Designer: Learning Design for the Digital Age. Always fun to connect with readers and fellow accidents.
- And finally, Vegas has amazing food – no doubt. AND I got to experience the Sphere (we didn’t splurge for U2 tickets, but we did see an amazing $90 movie! Hold on to your seats, it was awesome.)
Here’s a run down of the sessions I attended (and some I led!).
Morning Buzz – AI and Your L&D Role
Josh Cavalier
Josh is a big AI player right now in the industry,
definitely one to follow.
Don’t wait. Start now. Things are changing FAST. We don’t
know where things are going to be in 3 months, let alone a year. Everything is
changing at rapid pace, but we do know that AI is here and it’s disrupting
everything.
Lots of fear in the crowd – “what’s this going to do to my
job?”
Get a corporate policy in place.
Policy > Manifesto > Playbook
Manifesto – how your L&D team is going to use AI
Playbook – what tools you’re going to use and how – and
then meet on that playbook monthly (this idea actually came from Marcus
Bernhardt who was in the crowd – also an AI guy)
Opening
Keynote: Phil Rosenthal
Phil
Rosenthal was a producer/writer of the show “Everybody Loves Raymond”
and now has a show on Netflix – 7 seasons – “Somebody Feed Raymond”
where he travels the world and eats good food and makes people laugh.
Use your
constraints – working within your box can unleash your creativity.
The show
Raymond – they wrote it to be intentionally timeless. Keep out any time bound
references (e.g., you don’t mention Bill Clinton in the 90’s) and you make the
content RELATABLE.
The best
stories are those that are relatable. As a storyteller, satisfy your own
interests and be specific. When we are specific, people are more able to relate
to your story.
“Do the
show you want to do because we all get canclled.”
Creating a Robust
Company Learning Culture with Values-Driven Development
Jennifer Gilley,
Anuvu; and Amy Wheldon, Go1
Building a
values-driven learning culture at an org obviously needs to start with stated
values. Then follow a methodology:
- Build an authentic FOUNDATION
that centers on philosophy, values, and the employee experience. - Execute meaningful form – uncover
the opportunity, informed strategy, culture shift, feedback loops - Leverage Effective tools –
training and delivery, input and output
FOUNDATION
Anchor in your
beliefs and values.
Your org’s learning
philosophy informs your L&D strategy.
- Gather input from stakeholders
- Map out what learning looks like
- Write a philosophy statement
(who, what, why) - This is high level – no perfect
formulas for this - What does learning look like and
what does it mean in your organization?
Core values:
- How do you integrate these values
into your culture? - Incorporate these values to
inform training focus and to elevate the employee experience
The Employee
Experience:
- Journey mapping
- What does it feel like to work
here? - How can we create an environment
where people can truly grow? - Where can we partner across the
business as reinforcement?
L&D Program is
really about change management – a form to drive change. Think about:
- Opportunities
- Strategy
- Culture Shift
- Feedback Loops
Uncovering Opportunities:
- Use results from annual and pulse
engagement surveys - Look at performance metrics –
review SLAS – to identify gaps around skills/competencies - Get stakeholder feedback –
informal and formal to round out data points
Create an Informed Strategy:
- What are our biggest challenges?
- Prioritize based on those that
will have the greatest impact - ID groups that are most impacted
- Build from competencies
- Automate training
- “This is how the new L&D
program is going to support…”
Culture Stuff:
- Model enthusiasm – leverage
positive psychology, mirroring, cascade across and down all levels of
leadership - Overt communications
- Shared accountability – tracking
with targets to meet
Feedback Loops:
- Formal tools to get employee
feedback - Leverage managers for informal
feedback and formal - Leaders to demonstrate the new
behaviors (modeling) and do check-ins with teams
Tools:
Your LMS,
facilitators, surveys, what they called “elevators” (ways to elevate
the experience – whiteboards, polls, gamification of sessions).
For compliance
training – if you’re trying to actually change culture, think about how these
behaviors ladder up to your values.
For leaders/managers
– provide them with a check-in template (for meetings, conversations)
When change happens,
values usually don’t change. But execution does chage. Ask “How do we
define what it means to exhibit our values?”
The Hitchiker’s
Guide to AI
Matt Donavan, GP
Strategies
Start first with
what the work is you need to do (not the tools, etc)
Put up Enterprise
Guard Rails:
- Responsible Use of AI
- AI Policy
- Approved Tools
Guidelines for
responsible use of AI:
- Maintain privacy and security
(how identifying data being used) - Ensure safety and well-being
(provide guidance) - Embrace DEI (how well does the AI
work in different regions of the world 0 does it worry the same in Japan
and Brazil…. - Drive accountability (is the
content generated accurate, do you have the rights for copy and or images
being managed. Does it meet quality standards? - Convey Transparency: let people
know when you’re using it – don’t be ashamed, but open. - Promote Fair and Equitable: does
it bring people together. Does it drive equity?
GPT4 sitting in an
azure environment. Put all of your past RFPs on your system – point it to
meaningful data from historical RFPs. (but remember most org’s data is going to
be a hot mess – may require a lot of mopping up afterwards).
Your policy might be
“don’t put high-risk data onto that…”
Training Data Set
Through 2020 (?)
Constitutional
Training (how you set these ecosystems up).
Need both human
guidelines in place and the how we’re structuring machines
How can AI be used
to support a manager in the timeline completion of quality of performance
conversations?
Performance
Management Conversation Workflow….where could AI support this workflow?
- These are the tasks in the
workflow: Remember (to do it!), schedule, locate prior documentation, prep
or refresh, coaching or mentoring, conducting the conversation, complete
the documentation - Could you have AI help with those
sticky spots? AI could check if you’ve scheduled it. It could go out to
your system and get feedback from people, then summarized the people –
then it locates the docs for you, then brings in other data and info.
Morning Buzz – eLearning – what do people
really want and need?
Cammy Bean, Kineo
This was an informal conversation about learner needs – and
the reality of what’s going on in the real world. Individuals and orgs are at all levels of
maturity in terms of elearning. Lots of Storyline use. People leveraging
YouTube videos into course content (why recreate what’s already out there and
that people really want to see anyways – first thing most people do is Google
It or go to You Tube to learn).
Thursday Keynote:
The Five Mindsets of Innovation
Business leaders
need to be like jazz musicians
Adapt, navigate
uncertainty, situation awareness active listening, real-time collab and problem
solving, recovering quickly from
setbacks.
Mundita – venture capital firm with a soul (this is his VC company)
Everyday Innovations
- Start before you’re ready
Awaken the artist
within
How have I always
done it in the past. What would it look like if you did the opposite the Judo flip.
Savannah Bananas
Reduce risk through experimentation.
Lots of little ideas that you test – 15 minutes and 20 bucks fast. Cheap and
fast.
Create a To-Test
list – like a to do list. Keep a running list of things to try and test – the
mere existence of the list will boost your innovation. Do a shared list.
“What are we doing to test this week.” Inject an experimentation
mindset.
- Break it to Fix it.
Start with a blank page – Jerry Seinfeld throws out all
his material every year.
Let go to move ahead. A hallmark of
innovators. Release an existing system to discover a new one.
At Ben & Jerry’s when a flavor is
no longer successful, they physically/literally BURY IT. They have a graveyard.
They give it a champagne toast and read a eulogy. The let it go. It’s ok to let
go to move ahead. And they have a digital version on their website. It’s ok to
break it to fix it.
Try this….the business funeral. Look
to a system, process, technique, approach that yo’ve been doing for decades.
Give it a funeral and a cake and let it go. We spend too much time trying to
save the past.
Speedo – break it to fix it – it used
to be the fastest suit was smaller, smoother. Then they made a sharkskin
suit…full body.
Try this…speedo did the judo flip.
Instead of small and smooth it was large and not smooth.
Shatter conventional thinking
- Use every drop of toothpaste:
MacGyver It
Try this….print an image of a tv on
your bike box to reduce damage by 70%
4. SEEK THE UNEXPECTED
Before choosing A, B or C – think if
there’s an Option D or an Option X.
Option X is what are the obvious
answers and what are those far out answers, the unexpected approach.
Pittsburgh Children’s Hospital wanted
to create a better experience – they outfitted the window washers in super hero
costumes. Little innovations, big breakthroughs. This cost virtually $0 and
made a huge difference.
Bring those ideas to the surface. Find
the oddball idea that makes all the difference in the world.
- Fall seven times, stand eight.
Setbacks are part of
the process, not shameful – a learning opportunity. Each time, try something
new (don’t keep doing the same thing). Try a different creative approach.
100% chance with
jazz music that I’m going to screw it up. Confidence doesn’t come from
expecting to get it right, it’s from knowing how to bounce back.
Downtown Boxing Gym
(check it out)
Learning
Architecture Design: Anchoring ID/LXD in Business Transformation
Frank Nguyen,
Genentech (and formerly Amazon)
ID came about out of
WWII.
Robert Gagne, 1956
The Conditions of Learning – nine events of instruction – established the
beginning of instructional design.
Does training always
results in learning? No – sometimes.
Training is a tool –
sometimes a jackhammer – may be effective
but not always so efficient.
Walk out of a room,
you’ll forget 42% of what you heard; and then if in 3 months you haven’t used
it it’ll go down way more. Ebbinghaus Curve was 1859 (?) – but results continue
to be replicated.
Gagne’s nine events
have really served as the basis of ID.
You might sit in a
class and follow all the nine events, it might still be really boring.
Following conventional instructional design can be really dull.
- Early 90s – world
wide web! - 2005 Wikipedia and
Web 2.0 - 2005 YouTube
- 2007 iPhone
- “Mobile
Learning” people don’t even know what that is now – it’s just part of the
world.
So how do we design
around this new world?
Instructional Design
+ User Experience Design = Learning Experience Design – ID has to evolve to
encompass ALL these ways of learning and working.
So does Learning
Experience Design always result in learning? No – maybe sometimes.
Do Learning
Experiences always achieve Business Goals or Transformation?
Instruction Design
> Learning Experience Design > are we due for the next evolution?
What is the problem
with LD and LXD?
Are we focused on
learning outcomes or business outcomes? Remember, training/learning may not
always be the solution.
These don’t move at
the speed of the business – the business needs much faster solutions
Immediacy,
application, tying it back to the learner.
Tenet #1:
Sometimes learning
happens because of training, but learning happens all the time with or without
us.
Learning produces
business outcomes. Run a learning organization and not a training organization.
Jane Hart’s list of
top learning tools:
YouTube, Google
Search, Teams, ChatGPT….then things like Zoom went down….there are no
“training tools” on the top ten list – a user focused list. Where do
you go to learn? Learning is happening all the time, with or without us.
How are you pulling
in youtube etc into your learning architecture?
Tenet #2:
Learning is just one
leg of the tripod for change management and business transformation
Change management
model: ADKAR
- Awareness (Communication –
lighter weight, less frequency – training class is a jackhammer when you
need a screwdriver), - Desire (marketing & comms),
- Knowledge (training),
- Ability (workflow learning and
experiential learning), - Reinforcement (workflow learning,
incentives, performance management, marketing and comms)
How do you bring
value to people’s daily jobs…to drive real business transformation, we have to
think outside the training box.
Tenet #3:
Focus on learning
outcomes not business outcomes
Working Backwards –
Insights, Stories and Secrets from Inside Amazon (Colin Bryar and Bill Carr)
Solve real business
problems through learning, not just learning objectives
Working Backwards:
- Who is the customer?
- What is the customer problem or
opportunity? What problem does the customer need to solve? - What does the customer experience
look like? - Is the opportunity compelling
enough? - Build a roadmap, working backward
from the end-state.
You may need to have
a long perspective – it can take a long time to change.
“That’s where
we want to be and these are the iterative steps tto get there.”
Instructional Design
> Learning Experience Design > Learning Architecture Design (THIS is the current transformation)
Learning
Architecture Design = Much more high-level.
Learning Architect
Working backwards
- Who is the customer (learner)?
- What is the business problem (not
the learning problem)? - What will be different in the
business over next one, two, three years? - Design a learning architecture by
working backwards from the end state - Anchor marketing, comms, and
learning to business milestones or seasonality
Learning side will
need to be agile because the business may change.
Amazon fulfillment
center:
Jan – ramp down
Feb-May –
invention/continuous improvement
June – ramp up
July Prime Day
(created to be a test day for inventions)
Aug-Sept iteration
October ramp up
Nov-Dec peak
- Align
training/learning architecture with business seasonality
Think about:
- Business Milestone
- Marketing
- Communicate
- Workflow learning
- Training (last
resort)
Managers – consider
creating a 15 minute meeting in a box for an experiential learning exercise in
a team meting
If training/learning
is tied to your job level, pay, etc. you may pay attention/learn
https://frankn.net/DL23-WBT.pptx
Getting Started with Learning Content Anaytics
Cammy Bean, Joe Replin, Rodrigo Bolanos, Kineo
This was our session! Quick story – analytics is top of mind for L&D but it’s a low skill set on most teams. Many orgs are just trying to figure out how to get started and there’s a lot of overwhelm. We wanted to set the stage for why data & analytics are so crucial to L&D, share a framework for getting started with building these skills by looking more closely at learning content analytics, and share some examples and case studies from some of our work with different types of organizations.
Our deck is up on slideshare for your viewing pleasure.
Harnessing
the Power of AI to Develop Scenarios
Jennifer
Foster, Udutu
Using AI
tools to jump start your workflow and generate branching scenarios:
- Use ChatGPT4 ($20 a month) – to
generate list of questions to ask your SME about X topic.
- Scenario Builder Epic ChatGPT
(based on Cristy Tucker’s branching scenario work)
- Zoom just came out with an AI
tool last week – it transcribes your whole meeting for you and then throw
that into ChatGPT – “hey give me my action items from that call”
- Get around the uncanny valley
feeling about avatars by just telling your learners upfront – “hey,
this was created with AI”.
- Research show: AI created avatar
videos match human presented ones in effectiveness
Making Your L&D
Case by Storytelling with Learning Data
Eli Bendet-Taicher,
Wix
Think about the
story you want to tell – think about different ways to tell that data story –
two to three ways (each version will vary depending on where you put your hook,
your facts, etc.)
What do you want
your audience to…?
- Know – facts data
- Think – perception
- Feel – how do you want them to
feel about the problem - Do – what do you actually want
them to do after the session?
“Data in
L&D is a lot like sex….everyone is talking about it, but not many are doing
it, or even doing it well.”
First party data:
- What can I collect myself? (e.g.,
directly from your LMS if you’re in L&D, etc.)
Second party data:
- What kind of data do other
departments collect within the organization that I can leverage? - How can I get my hands on second
party data? - Second party data – sales,
revenue, engagement surveys, performance reviews, business roles
performance, quality control
Ideally, you want to
correlate first party with second data to show impact
- Reaction = first
party data - Learning = first
party data - Behavior Change
= first and second party data - Results = second
party data
ROI (financial
metric) vs. ROE (indicator of value
ROE – measurement of
qualitative and quantitative benefits – did you meet expectations?
Correlating L&D
Data with KPIS – the flow.
- What are your organizational
business KPIS - How does L&D support the
business KPIS - Derive your own KPI from business
KPIs - Collect data
- Correlate all data points
- What kind of story does it tell
you? - Communicate that story to
stakeholders
We remember numbers
when the numbers serve the story (he showed a pitch video and the guy shared a
lot of numbers) – when you tell a story with data, think about the story you
want to tell (replace your adjectives with numbers — a lot, again and again,
any, not a lot-huge, game-changer, surprising – can probably all be replaced
with numbers.)
Use infographics
Tell the data story
and PR your work.
Closing Keynote: Michelle Poler
Michelle Poler, author of Hello
Fears and survivor of her own “100 Days Without Fear Experiment”
This was a feel good
session that literally had the crowd dancing by the end. Michelle shared
stories of facing ones fears, being clear on what you want, and making a shift in
your life to live more authentically and courageously.
The Expo Hall
150+ vendors – biggest year ever
BIG booths like DevLearn has never seen before!
· Articulate
– showing off AI power that’s coming soon to a Rise course near you (upload
your content docs and press a button and voila – a course! Click a button to
translate your content. Includes Video Avatar creator).
· Regis
– showed off an AI driven branching scenario builder.
·
Note – both Articulate and Regis were a
bit smoke and mirrors as I suspect all developers were working until the day
before the conference to get things together. This is all happening in REAL
TIME.
· Big
LMS booths from Docebo, Schoox.
· SmartCat – for
AI driven translations
OTHER
And then there’s all the conversations in the halls and
over dinners and captured in passing.
Lots of interesting things happening with AI driven
chatbots and agents – this will change how people search for and access content
– looking for answers when they need it in the workflow.
AI will change our workflow – how we do our everyday tasks.
This will put huge pressure on content companies to reduce cost and speed to
market. We need to pivot fast.
It will also change how learners search for and consume
content – Learner Experience will be less about a big 45 minute course and more
about that chatbot….of course, formal learning/training will still exist, but
it’s going to be different.
Obviously, all the concern about regulations and ethics and
privacy in any conversation on AI. While I don’t mention that much in my notes,
it’s an ever-present thread (thus some of the fear).
Lots of fear, uncertainty – but also a lot of EXCITEMENT
about the possibilities.
See you next year, #DevLearn!
