[0:17] Excused
[0:17] all
[0:19] excused kelty
[0:22] excused pascal
[0:24] here
[0:25] vice chair to tone
[0:27] I am here
[0:30] adam chair here
[0:32] OK
[0:35] first up we've got4
[0:38] departments here to talk about
[0:39] to continue our conversations
[0:43] about Ai and so we're eager
[0:46] to get to them in the meantime
[0:49] we have invited the officeice
[0:51] offormation Technology to come
[0:54] forward and do a quick
[0:59] presentation on the reorg
[1:00] that has happened in the
[1:00] department
[1:04] so they're gonna present for you
[1:06] know if you can keep it to 10
[1:08] minutes that would be great. We
[1:09] may have a couple of questions
[1:11] but I'm gonna limit us to to5
[1:14] minutes so we can get into
[1:15] what's next the meat of what
[1:17] we're going to talk about is
[1:19] gonna be about45 minutes for
[1:21] hick puff and then I would ask
[1:24] CDLE to hopefully stick to 20
[1:26] minutes and then
[1:29] CDpS if they're in the room
[1:31] we'll try to do 20 minutes as
[1:34] well. and 10 will be an
[1:37] executive session assuming that
[1:39] senator Ball joins us or rep
[1:41] Kelty joins us. they've both
[1:43] RSVpd yes. so with that we will
[1:46] go ahead and get started.
[1:46] director Tunberg
[1:54] good morning. thank you for
[1:57] having us. Can somebody make
[1:58] page a presenter please.
[2:02] Thank you
[2:06] thank you senator Marchman as
[2:08] you said we are here to briefly
[2:11] provide an update. we just
[2:14] crossed our 90 days since the
[2:16] second part of our very large
[2:17] reorganization and reset
[2:22] ' s very well thank you
[2:26] great.ext slide please
[2:30] we've provided these materials
[2:31] to you so I'm going to go
[2:34] through them as you said very
[2:35] briefly happy to follow up on
[2:37] any questions. The first thing I
[2:38] want to start with is the
[2:39] grounding principles of our
[2:42] reset and our reorg that
[2:44] really guide the way we are
[2:46] working as a whiteT moving
[2:47] forward. there are a couple I
[2:49] want to flag for you. The first
[2:51] is starting with our users we're
[2:54] in service to Coloradans the
[2:56] taxpayers who need us to deliver
[2:57] value through technology
[3:00] Additionally we are focused on
[3:02] outcomes over process. we
[3:04] want to be delivering measurable
[3:05] value as the outcome
[3:10] thank you.
[3:14] umpresentative Pascal requested
[3:15] that we provide a brief update
[3:19] on the work in flight at the
[3:21] point at which we did the sort
[3:22] of phase two of our
[3:26] reorganization we had this
[3:28] was in the end of May more than
[3:31] 200 in flight projects or
[3:36] products or programs of those
[3:38] 197 continue
[3:42] uninterrupted. we have
[3:45] successfully completed25 an
[3:48] additional 16 are in close out.
[3:49] we have achieved the objectives
[3:51] and are either transitioning
[3:54] them to a long term operations
[3:56] and maintenance, finishing them
[3:56] out in some other way
[3:59] Additionally 10 have been
[4:02] strategically retired, canceled
[4:05] or combined with others in
[4:07] addition to not seeing
[4:10] interruption in our workloads we
[4:13] are seeing greater velocity,
[4:15] lower costs and
[4:20] faster time to completion as a
[4:20] result of the reorganization
[4:26] we as part of our reorganization
[4:28] we divided our org into three
[4:31] primary delivery functions
[4:33] supported by security and
[4:34] infrastructure additionally
[4:36] supported by our enterprise
[4:38] policy governance andans group
[4:39] and supported by our
[4:40] administration arm
[4:45] to talk through some of the
[4:47] value and outcomes we are seeing
[4:49] in each of those. In our
[4:50] delivery to Coloradan's layer
[4:53] last week we launched the mobile
[4:55] driver's license within my
[4:58] Colorado. this enables the 2
[4:59] million Coloradans who have a
[5:03] myolorado account to use their
[5:06] account to transit through TSA
[5:08] as well as gain access to
[5:10] federal buildings in a privacy
[5:10] preserving secure way
[5:15] Additionally we launchedignin
[5:17] with myolorado with our summer
[5:21] EBT program. this yielded a70%
[5:24] reduction in the time it took
[5:26] families to access their summer
[5:29] EBT benefit. This seems somewhat
[5:30] funny cause we went from a
[5:31] little bit more than 2 minutes
[5:34] to 25 seconds but this is the
[5:36] way that digital services should
[5:39] be delivered frictionless, easy,
[5:40] secure privacy preserving.
[5:43] sure
[5:44] rapascal
[5:48] thank you madam chair. So can
[5:50] you tell what is the summer EBT
[5:51] program and how is that
[5:53] different from like regular
[5:55] ongoing activity with that
[5:57] program Directorymberg thank you
[5:59] for the question. The summer EBT
[6:01] programme is a program that
[6:03] enabled when kids can't get free
[6:05] and reduced breakfast or lunch
[6:07] at school. it provides a cash
[6:09] benefit that they can use to
[6:10] ensure that they have proper
[6:10] nutrition and feeding throughout
[6:11] the year.
[6:12] the summer
[6:19] repascal and I'm also curious
[6:21] about the mobile driver's
[6:24] license I know that we already
[6:25] could get our driver's license
[6:27] through myolorado very useful
[6:29] when I lost my driver's license
[6:31] but how is this different? Is is
[6:33] being able to get through TSA
[6:35] with this a new thing or what's
[6:37] differentrectorunberg great
[6:39] question. yes. so the visual
[6:41] version the digital driver's
[6:42] license that you can use within
[6:44] Colorado is not always accepted
[6:44] in other
[6:47] states or by federal partners
[6:50] like TSA it was not you couldn't
[6:52] use it to travel which is a
[6:53] primary thing people need their
[6:55] I for so the mobile driver's
[6:57] license is a new technology
[6:58] standard. It was available
[6:59] within the Apple wallllet
[7:01] Samsung wallet, Google wallet
[7:04] but now we have a nonig tech
[7:05] choice for Coloradan's really
[7:06] meeting Coloradans where they
[7:07] are
[7:09] OK and I do see we have
[7:11] additional questions. You only
[7:12] have4 minutes left so I'm gonna
[7:14] give you your4 minutes back you
[7:16] figure out what you need to do
[7:17] with those and then we will get
[7:20] to questions. Fantastic. So in
[7:22] addition we on the delivery to
[7:25] agency side are primary
[7:26] customers are agencies who we
[7:27] deliver to and then on to
[7:30] Coloradans we are seeing
[7:31] significant direct agency
[7:34] savings we have reduced overhead
[7:36] in OitT operations costs for
[7:36] example
[7:41] CdPS is we project a $1.2
[7:43] million savings in fiscal year
[7:45] 27 by having a dedicated
[7:47] staffing model and reducing the
[7:48] overhead. You can look through
[7:50] these later but we see this
[7:51] consistently across all of our
[7:52] agencies
[7:56] on a delivery to state employees
[7:58] significantly reduced the manual
[7:59] effort of key functions like
[8:02] human resources or constituent
[8:03] services at the governor's
[8:06] office on security initiatives
[8:08] we over the last few months have
[8:11] had an uptick in external
[8:13] international actors attempting
[8:15] to attack our infrastructure.
[8:17] The Iranian water system in
[8:20] attack jill Frazier ourisso
[8:22] together with our security team
[8:22] provided crucial
[8:25] intelligence that supported that
[8:27] response. Additionally we're
[8:29] seeing North Korean threat
[8:31] actors attempting to gain
[8:33] employment and access our
[8:34] systems
[8:37] drafted and released new HR
[8:38] processes that are keeping North
[8:39] Korean actors out of our
[8:42] essential systems more there on
[8:43] platforms and infrastructure
[8:46] we've already achieved $6
[8:49] million in platform savings via
[8:50] direct negotiations with our
[8:51] vendors we are deeply grateful
[8:53] for them.dditionally we've
[8:55] enrolled in GovRamp which
[8:58] reduces duplicating assessments
[9:00] for the services we now leverage
[9:01] clear third party risk
[9:03] assessments and continuous
[9:04] monitoring
[9:07] reducing the cost increasing the
[9:08] security of our platforms and
[9:08] infrastructure.
[9:12] on hiring we have received and
[9:15] processed nearly 3000
[9:17] applications for the various
[9:19] roles we have posted we've
[9:21] hired88 new individuals
[9:23] including 1f returning OIT
[9:25] employees
[9:30] arepod hiring this is our
[9:31] dedicated agency support
[9:37] or72% complete. we've hired55
[9:41] of76 targeted roles and e of our
[9:42] agencies have a
[9:47] full staffing so really making a
[9:48] haste on that ensuring we're
[9:49] providing value to our agencies.
[9:53] We just completed our 6th month
[9:56] of monthly employee checkin
[9:56] survey
[10:00] one of the questions we ask our
[10:02] employees is how are they
[10:05] feeling about the change.
[10:07] this is about the reorg and we
[10:08] ask are you on board?
[10:10] are you cautious to have
[10:12] questions? are you stuck?
[10:15] starting in March we saw that
[10:17] the cautious group really led
[10:22] the way in July our numbers
[10:24] declined this is right after we
[10:25] announced the layoff as well as
[10:27] fiscal end of year a lot of work
[10:29] and a lot of friction but in
[10:31] August a couple of weeks ago we
[10:33] really see these numbers switch.
[10:36] we see for the first time a
[10:37] really significant increase in
[10:38] those people who are saying
[10:40] they are on board this change.
[10:42] they are seeing value and
[10:43] benefit and those who are
[10:44] cautious to climb
[10:47] stuck remains the same and we
[10:49] will continue to help those
[10:51] folks move along. Last week we
[10:53] launched organizational roadmaps
[10:54] which I have shared with you.
[10:55] it's the first time in my
[10:56] knowledge OIT has had public
[10:59] roadmaps and a public technology
[11:01] strategy. I welcome you to look
[11:02] at those at your leisure.
[11:04] I believe that is it
[11:10] and40 seconds left brilliant way
[11:12] to go. I'm excited to see
[11:13] where those trends go. are you
[11:17] going to continue to do those
[11:19] that employee change readiness
[11:21] survey. Yes, thank you for the
[11:23] question and we'll in addition
[11:24] to continuing it each month
[11:25] we'll continue to share the data
[11:28] with you. Obviously it is a one
[11:30] point in time data collection
[11:33] not a trend yet but we hope we
[11:34] continue. Same great. I just
[11:35] want to acknow
[11:38] led ge that we have the rest of
[11:38] our committee here
[11:43] we have well most we have
[11:46] representative Kelty and senator
[11:49] Ball here. we have
[11:51] representative to tone with her
[11:52] hand uppresentative to tone
[11:57] thank you very much. thank
[11:58] you for the presentation.
[12:02] I think some of them might
[12:07] colorado things are cool I
[12:08] these are kinds of things that
[12:10] you know we had never heard that
[12:11] you were working on before and
[12:12] I'm kind of
[12:15] wondering why you were
[12:17] prioritizing these kinds of
[12:19] more vanity projects
[12:22] then focusing on some of the
[12:25] other issues that seem to be
[12:27] more pressing at this time as
[12:29] you pointed out especially the
[12:31] cybersecurity threats being that
[12:32] we have a pretty
[12:35] difficult situation with our
[12:37] cybersecurity in addition
[12:40] I have heard through the
[12:44] grapevine that there are many
[12:46] employees that are at their
[12:47] breaking point
[12:51] and they're being overworked and
[12:53] maybe that's why you're
[12:55] getting such great performance
[12:57] and cost reductions because
[12:58] you're working people to the
[12:58] bone
[13:03] and I'm worried about those
[13:04] employees who are valuable
[13:05] assets to the organization
[13:09] not wanting to be with the
[13:10] organization anymore because
[13:10] they're being
[13:11] broken
[13:16] and overworked. So how tell me
[13:16] about that and and
[13:19] tell me what you're gonna do
[13:20] about it
[13:24] OK so I'm just going to take a
[13:26] pause. We have had some
[13:30] conversations with OlLS and
[13:31] Ed Decheco to try to understand
[13:35] what the authority of the JTC is
[13:39] in in talking to a number of
[13:41] these employees we do think
[13:43] there are a couple of nuggets of
[13:45] wisdom that we'd like to add to
[13:47] the quarterly report that we
[13:49] would either run a bill or talk
[13:50] to you about but
[13:55] in terms of statutory authority
[13:59] OIT is called out in
[14:03] 2437.51057 that they are the
[14:05] ones in charge of all the Hr
[14:08] stuff so we are going to be
[14:12] focused on the IT stuff but
[14:14] we are not going to be able to
[14:18] oversee the HR things. That said
[14:20] just one second please
[14:23] I would I'm gonna give you
[14:25] the floor to respond to whatever
[14:27] you would like to relate to that
[14:29] question but knowing that we are
[14:31] not going to be in a position to
[14:33] take any kind of a role because
[14:35] statutorily we don't hire fire
[14:37] we don't get involved in the Hr
[14:40] of of things so I'm gonna go to
[14:41] directorunonberg and then I'll
[14:42] go back toresentative Taonene.
[14:45] thank you madam chair and thank
[14:47] youpresentative Taonene for the
[14:49] question. As you pointed
[14:51] outpresentative to tone are
[14:52] people and employees and
[14:54] colleagues at OIT are our
[14:56] greatest asset. They are the the
[14:58] mechanism by which we deliver
[15:00] value to agency partners to
[15:01] state employees and to
[15:04] Coloradans as I pointed out
[15:06] we are seeing more and more
[15:09] colleagues be excited and
[15:11] invigorated and have positive
[15:12] feedback regarding the reorgan
[15:14] ization
[15:17] I know that it is a change.
[15:20] change is hard and we are doing
[15:22] a lot of work to support
[15:23] employees through the change
[15:25] that looks like significant
[15:27] amounts of training more than
[15:30] we've ever offered very clear
[15:33] performance expectations so that
[15:34] people know what they're
[15:36] expected to do and how they're
[15:38] supposed to show up in this
[15:40] model. I appreciate again
[15:43] changes hard and I do not
[15:44] minimize that
[15:47] and we're seeing people excited
[15:48] and the feedback is strong.
[15:53] very good and we are out of time
[15:55] but I am going to have rep to
[15:58] tone you can ask one last
[15:59] question and we'll see if we
[16:01] can't get like a really quick
[16:02] response and then we're going to
[16:04] move on to the business at hand
[16:04] today.
[16:07] we'rep tommy my first question
[16:09] my first question wasn't
[16:11] answered so I would like that
[16:13] answer but I think that the fact
[16:17] that you know employment was
[16:18] brought up by Misshomberg
[16:22] in the presentation and the
[16:24] reorganization and loss of
[16:24] employment
[16:28] is something that was not
[16:31] given anybody was given a heads
[16:33] up about and that affects how
[16:36] the organization functions and
[16:38] that reflects on our oversight
[16:41] duty of how the work is getting
[16:42] done and how the taxpayers money
[16:44] is being spent so I think that
[16:44] it is very relevant
[16:46] to what's going on
[16:49] but but that's what I would like
[16:53] to know and why why when I asked
[16:55] for questions that the
[16:56] questions are not being answered
[16:59] in a timely manner of've sent
[17:02] several emails and why is the
[17:02] legislative liaison
[17:06] not responding to my questions
[17:06] when I
[17:10] ask her questions on text. there
[17:11] there's there's a
[17:15] of a way of trying to
[17:17] get information to not get into
[17:19] my hands so it makes me very
[17:21] suspicious of what's going on
[17:24] with OIT over the last
[17:26] several months prior to you
[17:28] getting into power here and with
[17:31] you there and that is a very big
[17:32] concern of mine
[17:35] and I wanna know what kind of
[17:36] transparency you're going to
[17:37] give to us when we ask for
[17:38] questions.
[17:43] and I will just kind of share
[17:45] what what the conversation that
[17:47] we had you guys are we're
[17:49] going to be done with the
[17:50] conversation at this point, but
[17:51] we are going to be adding to the
[17:53] quarterly update just as a heads
[17:57] up. the number and cost and
[18:01] project number and costs and
[18:03] projects by contractor versus
[18:07] employee in terms of how you
[18:08] employ folks and then
[18:11] adding all that information
[18:13] aboutervicehub just the projects
[18:15] that are ongoing so I think that
[18:16] will actually address a lot of
[18:19] it and I'm hopeful that that is
[18:21] going to keep us in our lane in
[18:24] terms of ledge liaison we
[18:26] need to make sure that we stick
[18:29] to the formal channels of
[18:31] communicating especially
[18:32] during interim. I know it's a
[18:34] really busy time. I'm struggling
[18:36] myself to keep up with text
[18:38] messages so I would just say
[18:38] that a really
[18:43] good norm committee team is
[18:47] if we could stick to our
[18:49] official forms of communication
[18:51] through email. and please
[18:53] feel free to copy any and all of
[18:56] us on that so with that we're
[18:57] gonna move along and we'll
[19:01] welcome hickpuff to the table.
[19:02] Come on up. thank you.
[19:45] Wonderful thank you. yes
[19:45] please go ahead.
[19:50] marchman my name is Ralph Cho.
[19:50] I'm blessed to be the chief
[19:52] operations officer for
[19:53] healthcare policy and finance.
[19:56] I'm also ecstatic to be one of
[19:57] the senior executive team
[19:59] members in charge of the AI
[20:01] governance over at Hickpuff. I I
[20:02] will tell you for someone who
[20:03] has been in the business world
[20:04] long enough to remember when
[20:05] there weren't PCs
[20:09] the the advent of artificial
[20:11] intelligence and what it can be
[20:13] for our organization to help the
[20:15] efficiency, the effectiveness,
[20:17] the delivery of services to our
[20:19] members the interactions with
[20:21] our providers are all very very
[20:23] exciting things but they come
[20:25] with a real price in making sure
[20:26] that those things are done
[20:29] prudently, intelligently
[20:32] today we've got a number of
[20:32] folks we're going to shuttle in
[20:34] and out a number of folks we
[20:35] brought our greatest cast of
[20:36] characters
[20:39] with us so you know my job was
[20:41] to to really to just introduce
[20:42] introduce the theme and the
[20:45] enthusiasm that we have as we
[20:46] all know is one of the largest
[20:48] spends within the state wherever
[20:50] we can find ways to make things
[20:51] better, faster, smarter, more
[20:54] intelligent, more effective we
[20:55] need to do that but we need to
[20:56] do it with guidance and
[20:58] I'm gonna ask Shaland to
[20:59] introduce himself since he's
[21:01] going to cover our governance
[21:02] processes to give you a
[21:03] foundation and then we have I
[21:05] know a number of questions that
[21:06] you all have asked across a
[21:08] number of systems and we will be
[21:09] shuttling the folks through to
[21:11] be able to answer those specific
[21:12] questions on the specific
[21:14] questions and at some point here
[21:17] I will defer one of these
[21:19] seats or my seat Nina Mac is
[21:21] in the in the building and
[21:23] you you know part of this is
[21:25] CBMS and so don't don't mind us
[21:26] while we shift and move seat
[21:26] s
[21:30] sounds great and just for
[21:33] timekeeping um45 minutes so
[21:34] however you guys want to maybe
[21:36] this first one tell me how many
[21:38] m or whatever you want to do but
[21:39] if you could keep time, I just
[21:41] got45 minutes on my clock. Very
[21:43] good. Who would like to go? Very
[21:44] good. thank you.
[21:47] thank you madam Chair members of
[21:48] the committee. I'm Shaon Weiner.
[21:50] I'm the department's deputycFO
[21:52] overseeing compliance and
[21:52] enterprise risk.
[21:55] and I want to follow on with
[21:58] what Ralph said Ai is a brand
[21:59] new thing. there's a lot of
[22:01] potential in it but hickpuff
[22:03] wanted to make sure that when we
[22:05] approach this we did it right.
[22:06] we don't want to do Ai just for
[22:09] the sake of Ai we want to do Ai
[22:11] right and so we started from a
[22:12] compliance mindset
[22:14] we were one of the first
[22:17] agencies here or almost anywhere
[22:19] at some multi state conferences
[22:21] we've been asked to tell them
[22:22] how we approach governance
[22:23] because we were some of the
[22:25] first ones out of the gate with
[22:27] a strong governance structure
[22:29] and that governance structure
[22:31] uses a couple components. The
[22:33] first off we built an Ai
[22:34] business governance committee.
[22:35] so not only do we have the
[22:37] technology governance that we
[22:39] get with our partners at OIT but
[22:41] we said we don't want to just
[22:42] know what technologies are allow
[22:43] ed we want to know what
[22:46] technologies we should use and
[22:46] how we should use them.
[22:51] and so before any Ai tool rolls
[22:52] out we did have some that were
[22:53] in prior to this process that
[22:55] had gone through OIT approval
[22:57] but for the anything coming out
[22:59] now it goes through a governance
[23:01] process where we have a
[23:03] crossfunctional team. We have
[23:04] legal representatives we have
[23:05] privacy representatives we have
[23:07] technology representatives we
[23:08] have bias and ethics
[23:09] representatives we have program
[23:11] representatives and we want to
[23:12] make sure that we're coming at
[23:14] it from every perspective before
[23:15] we do that. We also have
[23:16] executive leadership myself,
[23:17] Ralph
[23:19] ar ish are on that committee
[23:20] as well
[23:23] Now this committee doesn't
[23:25] bypass other rules. We didn't
[23:26] want to set a separate process
[23:29] in place this focuses on is the
[23:31] Ai the right thing? So we still
[23:32] go through our normal OiT
[23:33] approvals we still get our
[23:34] support from our IT partners we
[23:36] work very cooperatively with
[23:37] them when we have our initial
[23:39] meetings with requesters they're
[23:40] sitting in the room right next
[23:42] to us asking questions too to
[23:43] make sure that we all have the
[23:44] answers we need.
[23:46] we also make sure we do budget
[23:47] approval just like we would for
[23:49] any other technology.hings have
[23:51] to be procured they go through
[23:53] the standard procurement process
[23:54] we get cybersecurity reviews, we
[23:56] get privacy reviews we get all
[23:57] of those things. So the bottom
[23:59] line of this we bring in the
[24:02] experts we need to make good
[24:03] decisions about the use of the
[24:04] tools that we have.
[24:08] on the other side of that we
[24:09] also want to make sure that we
[24:11] have the people there because Ai
[24:12] tools are just that they're
[24:14] tools they help people do things
[24:15] more efficiently. they help
[24:18] people do better things but they
[24:20] never replace people. the people
[24:23] are what do the work and so we
[24:24] designed this as a support
[24:26] structure to help those people.
[24:28] anything that comes out of Ai
[24:30] has to be reviewed whether it's
[24:30] a direct
[24:33] piece of output from a
[24:35] generative Ai we expect that the
[24:36] person getting it they're
[24:37] ultimately responsible for what
[24:39] it says and if it says something
[24:41] wrong they're held to
[24:41] accountable to it the same way
[24:43] as if they'd written something
[24:45] wrong on their own so they're
[24:47] responsible for that. on the
[24:48] other side sometimes we have
[24:49] systems that are using these
[24:51] processes we want to make sure
[24:52] that we have auditing in there
[24:53] that we're sampling that we're
[24:54] testing these things to make
[24:54] sure that the outputs are right
[24:59] we also make sure that we
[25:01] have privacy and security
[25:03] compliance. It does not get PHI
[25:05] unless it has a HIPA BA just
[25:06] like any other system out there.
[25:09] we make sure that we have those
[25:10] things in place before anyone
[25:12] can use it for that will
[25:13] restrict access we'll do things
[25:14] like that to make sure that we
[25:15] don't have those things happen.
[25:18] we also want to make sure that
[25:20] if there is any unapproved
[25:22] unsecure anything like that we
[25:24] have a strict prohibition
[25:24] against those things we have
[25:28] prohibited tools because they
[25:29] automatically join meetings
[25:31] without the person present. We
[25:32] didn't think that was
[25:33] appropriate and so we prohibited
[25:34] those and we've worked with our
[25:35] partners at OIT to lock that
[25:36] down so that people can't
[25:38] download those things. so we've
[25:40] taken those steps to make sure
[25:41] that we're doing it right and
[25:43] that's the biggest part that we
[25:44] have with our governance
[25:45] structure and so with that I'll
[25:48] hand it over to antoinette for
[25:50] one of the first case studies of
[25:50] one of the tools that we have
[25:51] approved.
[25:53] yes you may begin
[25:56] right
[26:00] good morning madam chairirre
[26:02] representatives and senators I'm
[26:04] Antontte Taranto, the chief
[26:05] customer officer for the
[26:06] department
[26:09] I don't know if this is close
[26:10] enough
[26:15] so I often think about
[26:16] government
[26:16] collecting taxes
[26:19] closer
[26:23] thank you. let's do that. OK. So
[26:25] I often think about government
[26:27] collecting taxes in two ways
[26:31] time and dollars. So my job is
[26:32] to shrink the time
[26:35] and medicaid is complicated.
[26:37] Healthir Colorado has more than
[26:39] 30 different benefit programs
[26:41] with services delivered through
[26:42] multiple partners and processes.
[26:45] A member may have a simple
[26:46] question
[26:50] and not knowing where to start
[26:53] what terminology to use or what
[26:55] information applies to them is
[26:56] is difficult
[26:58] so we added a conversational
[26:59] assistant to the Healthir
[27:01] Colorado mobile app. It's
[27:03] available 24/7 and members can
[27:05] answer questions in everyday
[27:06] language.
[27:09] the role of AI is intentionally
[27:12] narrow. It uses more than70
[27:13] approved healthalthir
[27:16] Coloradoours to generate plain
[27:17] language responses.
[27:21] and importantly this is just an
[27:23] informational tool. it does not
[27:25] determine eligibility, approve
[27:26] or deny services or coverage or
[27:27] make clinical decisions.
[27:31] and we regularly review
[27:35] the questions members are asking
[27:36] in addition to the answers being
[27:36] generated.
[27:41] and we're on slide two great
[27:45] so now that members are using
[27:46] it we're starting to see what
[27:46] this can tell us.
[27:49] Members have asked roughly
[27:53] 10,000 questions satisfaction is
[27:56] about72% and we're seeing89%
[27:59] accuracy. Most sessions are just
[28:00] one question.
[28:03] and to me that last piece really
[28:05] matters. Success isn't getting
[28:07] someone to sit there and have a
[28:08] long conversation about
[28:08] Medicaid.
[28:11] it's helping them get what they
[28:12] need and get on with their day.
[28:16] we're also learning from what
[28:16] they're asking
[28:19] The assistant is able to answer
[28:21] 62% of questions without
[28:22] directing the member elsewhere.
[28:27] for the remaining 38% we
[28:28] intentionally designed
[28:31] Ai to guide the member to the
[28:32] appropriate partner.
[28:37] and 30 of that 38% are going to
[28:39] counties for eligibilitylated
[28:41] needs which I think we all
[28:43] probably knew that but now we
[28:45] have information that confirms
[28:49] So technology is doing more than
[28:51] answering those questions it's
[28:52] showing us where the friction
[28:52] is.
[28:55] and lastly
[28:58] reiterating what some of what
[28:59] Shain has said
[29:03] before each use we tell members
[29:05] to not enter their protected
[29:07] health information or their
[29:09] personal information. If they do
[29:11] there are guardrails in place
[29:12] designed to remove that
[29:12] information.
[29:14] and that's all I have for you.
[29:14] thank you.
[29:18] very good rett Pasco.
[29:23] thank you madam chairir. so
[29:24] my question is about where you
[29:28] said89% accuracy so I first of
[29:29] all I want to double check is
[29:30] this
[29:32] the scenario where you've got
[29:33] someone who works for the
[29:35] department that's going to
[29:38] review it before it goes to the
[29:41] member or is it automatically
[29:43] going straight to the member and
[29:44] then
[29:46] what is the process for
[29:50] dealing with the other
[29:54] they can't do math 11% Miss
[29:54] Toronto.
[29:57] thank you madam chair andres
[30:00] representative. yes, I was
[30:02] hoping you'd ask that question.
[30:05] so we designed the Ai for that
[30:09] 11% to refer people to our
[30:11] member contact center. so it's
[30:13] not technically an inaccurate
[30:14] answer but when it doesn't know
[30:17] the answer it refers to our
[30:20] contact center. so what we've
[30:23] discovered in that 11% are
[30:24] times where
[30:27] people are asking for things we
[30:29] didn't train it for. for example
[30:31] people ask for explanation of
[30:35] benefits. or it's asked
[30:38] members often ask for pharmacy
[30:38] bin numbers
[30:42] things that we don't have in
[30:45] Medicaid. so we have to then we
[30:46] are learning from those
[30:48] questions and then we go back in
[30:49] and we train the Ai to say
[30:51] Medicaid doesn't have a
[30:52] pharmacybI number you use your
[30:55] state ID which consists of so
[30:57] those are the errors and again
[30:59] back to when we do identify
[31:01] those errors it is automatically
[31:04] referring folks to our member
[31:04] contact center.
[31:11] Yes, hett Pasco were you talking
[31:12] about one particular one of
[31:13] their programs or were you just
[31:15] talking in general about their
[31:16] errors. I just wanted to
[31:16] clarify.
[31:21] sorry sorry I think here
[31:22] we're talking about maybe this
[31:23] is what we should clarify also
[31:27] this is where members of are
[31:29] going to the website and asking
[31:33] questions and I I understand
[31:36] that the areas where the AI
[31:37] doesn't know the answer is that
[31:39] how we're defining it?rrect OK.
[31:41] correct and then you can go
[31:43] ahead and follow up. I know you
[31:45] had another question that was
[31:46] was trying to
[31:47] so
[31:50] is there any way that you guys
[31:53] are looking for like as they
[31:55] call it hallucinations like they
[31:57] give a very confident wrong
[31:59] answer and are we looking for
[32:02] that and and how and what's
[32:05] going on thereto yes madam
[32:09] chairirpresentative, yes I
[32:11] personally am looking at every
[32:13] single question and response. I
[32:15] am not seeing hallucinations
[32:17] where things are just flat out
[32:22] wrong we have trained it to
[32:23] specifically look at our
[32:28] documents the se0 and
[32:31] those guard rails are very tight
[32:34] so when it's not sure and it
[32:37] can't find it in that library
[32:38] it's saying you know I don't
[32:41] know thisact our staff at the
[32:42] member contact center.
[32:44] yeah
[32:45] pascal
[32:50] sorry
[32:53] so just to follow up, you
[32:55] guys have your own information
[32:57] set and it's your own sandbox
[32:59] it's not going out on the
[33:01] internet and finding random
[33:02] stuff or anything like that and
[33:04] if it can't find it in that set
[33:06] of data then it just says I
[33:08] don't know go to a human
[33:10] basically yes exactly
[33:13] Perfect. OK.
[33:17] is there any way
[33:19] I know we're giving a warning
[33:21] that says don't enter
[33:23] HIPparotected kind of stuff but
[33:28] since it isIPparotected data
[33:31] that people could be feeding in
[33:34] how are we separating that? I
[33:35] know I heard you say you're
[33:37] looking at every response that
[33:39] comes in. you would be the one.
[33:43] are there non clinical staff are
[33:44] are there folk like that would
[33:44] probably be
[33:45] you you're probably a non
[33:47] clinical staff. I'm not sure
[33:51] who's looking at HIPAda. Is that
[33:53] problematic or is there any way
[33:55] just to reject and identify
[33:57] hiAda like do you know what I
[33:59] mean? Do you understand what I'm
[34:01] saying? yes madam chair thank
[34:05] you. so I have yet to see hey
[34:06] I have this rash
[34:12] what do I do? what I am
[34:14] seeing are things like is this
[34:15] covered is that covered and
[34:18] those are in my mind general
[34:19] questions. now you can connect
[34:21] dots saying oh maybe this person
[34:24] needs chiropractic care but
[34:25] they're asking are these things
[34:27] covered? I have yet to see
[34:30] someone coming into it saying
[34:32] hey I have this going on what do
[34:36] I do? we also have our
[34:36] nurse advice line as
[34:39] one of the go to's
[34:43] for the responses if we suspect
[34:45] people are asking or wanting
[34:48] health information or you know
[34:48] clinical guidance
[34:55] thank you madam chairir. and one
[34:57] additional thing with that
[35:01] the system in which this exists
[35:03] already has a HIPA BA in place
[35:05] so we have our legal protections
[35:07] in place in addition to the
[35:09] functional protections. What's a
[35:11] BA Oh sorry a business associate
[35:16] agreement it's the forms it's
[35:17] the addition to the contract
[35:19] that allows us to provide PHI to
[35:22] a contractor. Got you and that
[35:22] is legal from C
[35:25] M S too. that was another
[35:27] question I had that's fine. I'm
[35:29] sure they're fine with sending
[35:31] that data. OK.
[35:33] any other questions
[35:38] OK. we might have saved time on
[35:41] your guys' so very good. thank
[35:43] you very much. We will move on
[35:44] to the next segment.
[35:48] good morning madamir and members
[35:49] of the committee. I'mco Libloom.
[35:51] I'm the provider relations and
[35:52] fiscal agent operations
[35:52] divisionreor
[35:55] at the department of
[35:56] healthalthcare policylicy and
[35:59] financing. thank you for the
[36:01] opportunity to discuss our use
[36:03] of artificial intelligence
[36:04] within the Colorado Medicaid
[36:07] provider call center. I want
[36:08] to begin by providing some
[36:10] contents about what the
[36:13] technology is, why we use it,
[36:15] and what it does not do. the
[36:16] department contracts with Optum
[36:17] to operate our provider contact
[36:19] center and as part of that
[36:20] service we use the core Ai
[36:22] platform to provide an enhanced
[36:22] conversational selfservi
[36:26] option for medicaid providers
[36:26] who contact call center.
[36:30] Our goal in implementing this
[36:31] technology was pretty pretty
[36:32] straightforward which was to
[36:33] improve
[36:35] service delivery to
[36:37] providers. provider's contact
[36:39] us with a range of of questions
[36:41] and many of those are routine
[36:42] inquiries where information the
[36:43] provider needs already exists
[36:46] when that within hickpuff's
[36:49] Medicaid system core AI allows
[36:50] us to to make that information
[36:52] more readily available through
[36:54] an automated conversational
[36:55] interaction while preserving
[36:57] live customer service resources
[36:59] for inquiries that require
[37:02] additional assistance. So
[37:02] when a provider calls the
[37:03] technology uses artificial
[37:03] intelligence
[37:05] to recognize the intent of the
[37:07] provider's request and determine
[37:08] the appropriate predefined
[37:10] conversational flow. the
[37:12] system can query Hickpuff's
[37:13] existing medicaid data and
[37:15] provide information responsive
[37:16] to the provider's inquiry
[37:18] and when an inquiry cannot be
[37:19] appropriately resolved through
[37:21] selfservice it's transferred to
[37:22] a live customer service agent.
[37:25] an important distinction is that
[37:27] artificial intelligence and
[37:29] automated decision making are
[37:31] not synonymous. Cory eyes and an
[37:33] ai enabled service interface and
[37:36] it's not a medicaid decision
[37:36] making system.
[37:39] I cannot approve or deny a
[37:41] claim it can't change or adjust
[37:42] a claim. it cannot determine
[37:44] payment amounts, approve or deny
[37:45] provider enrollment determine
[37:47] provider status or make member
[37:49] eligibility determinations.
[37:51] Those decisions continue to be
[37:52] made through Hickpuff's
[37:54] established medicaid systems
[37:55] and business processes.
[37:59] Core Ai can retrieve and
[38:01] communicate information about a
[38:02] decision that has already been
[38:04] made. for example it may provide
[38:05] a provider with information
[38:07] about the status of a claim
[38:09] but communicating the outcome of
[38:10] a decision made by Hickpuff
[38:12] system is very very different
[38:12] than making that decision
[38:13] itself.
[38:17] also want to distinguish between
[38:19] Hickpuff's implementation for
[38:22] some of the degenerative Ai
[38:23] technologies that we've received
[38:25] significant public attention
[38:26] core Ai technology platform is
[38:28] capable of supporting generative
[38:29] Ai however the department's
[38:31] implementation of the provider
[38:33] call center does not use
[38:35] generative Ai or large
[38:36] language modules
[38:39] The system does not
[38:40] independently form formulate
[38:41] answers or interpret medicaid
[38:43] policy.stead it uses intent
[38:45] recognition follows
[38:46] predefined conversational flows
[38:48] and retrieves information from
[38:50] HiPuff's existing system. That
[38:51] distinction is also important
[38:54] when considering the risk of Ai
[38:55] hallucinations which you
[38:57] talking about in your last
[38:59] conversation our tool is not
[39:00] independently generating
[39:01] information in response to
[39:03] provider questions it's
[39:04] responding within a fine
[39:06] conversational processes and
[39:06] retrieving information directly
[39:09] from hickpuff's medicaid system.
[39:13] it does not create or infer
[39:14] information beyond what is
[39:15] available through those
[39:17] processes and systems. data
[39:19] production protection is another
[39:21] important consideration whenever
[39:23] technology interacts with
[39:24] Medicaid information
[39:25] the platform does not retain
[39:29] provider or medicaid data beyond
[39:31] the active transaction limited
[39:32] operational logs are maintained
[39:34] for up to 90 days for security
[39:37] audit troubleshooting and
[39:39] support purposes and the data
[39:40] associated with the tools
[39:41] maintained within the
[39:42] continental United States.
[39:45] Finally the human component
[39:46] remains an important part of our
[39:48] service model. Coreyi is
[39:49] intended to enhance customer
[39:51] service not eliminate or replace
[39:53] human assistance routine
[39:54] inquiries can be handled
[39:57] efficiently through 24x7
[39:59] selfservice allowing live
[40:00] representatives to focus their
[40:01] time and expertise on
[40:03] interactions that require
[40:05] additional research or
[40:07] assistance and when an ur
[40:08] inquiry cannot be appropriately
[40:09] resolved through the automated
[40:11] interaction it's transferred to
[40:12] a live representative and
[40:15] handled through the department's
[40:16] established processes.
[40:17] Ultimately we view this as
[40:19] targeted and deliberately
[40:21] limited use of artificial
[40:24] intelligence. we're using AI
[40:25] for what it can do well in in
[40:28] this environment which is
[40:29] understand the nature of the
[40:30] provider's request to guide the
[40:31] interaction and help providers
[40:33] quickly and efficiently access
[40:35] information that already exists
[40:36] within our systems. we have
[40:38] not delegated Medicaid decision
[40:40] making authority to artificial
[40:41] intelligence and that
[40:42] distinction between using
[40:45] Ai to improve access to
[40:47] information and using Ai to make
[40:49] consequential decisions is
[40:51] fundamental to how we
[40:52] implemented this
[40:57] and and you'll see up on the
[40:58] slide just for
[41:01] comparison purposes our core
[41:03] AI product is about 2700
[41:05] interactions per month and we
[41:07] take about m probably 1f to
[41:09] 16,000 live live provider calls
[41:13] so with that if there's any
[41:14] questions I'd be happy to
[41:14] to answer them
[41:19] right and just so I understand,
[41:21] do is someone else gonna be
[41:23] presenting about the Olivia
[41:25] programme. OK, we only have
[41:28] about4 minutes left. I do see we
[41:31] have some questions. let's go
[41:32] ahead and take the questions
[41:36] but we'll try to be quick with
[41:38] them. OK senatorball
[41:43] thank you madamir and thanks for
[41:44] the presentation
[41:47] the slide has improved customer
[41:49] experience how do you measure
[41:51] that and is that improved over
[41:55] measurements when people talk to
[41:56] actual people
[42:00] thank you for the question
[42:04] Lindblom thank you for the
[42:04] question.
[42:09] so we do have another
[42:10] piece of new technology that we
[42:11] implemented with with our
[42:13] call center implementation and
[42:15] that was a survey tool so every
[42:16] provider is able to
[42:20] provide feedback on on that
[42:22] transaction whether or not it
[42:23] was through core Ai or from
[42:27] talking to a a a person on
[42:29] the phone. we started off
[42:33] with about a score of4.5 out of
[42:35] five when we first went live and
[42:37] in May of of 2025 and we're up
[42:41] to about a4.8 average score so
[42:43] we we have seen you know the
[42:47] satisfaction of the providers
[42:48] go up in the in the last year
[42:51] and a half and it also gives
[42:52] them a chance if they don't need
[42:54] to talk to a a a person just
[42:55] very very much like it would be
[42:57] if you or I called our bank and
[43:00] needed to know what is our
[43:02] account balance we didn't want
[43:03] to talk to somebody we could get
[43:04] that information through
[43:04] automated means
[43:06] Thank you
[43:09] a repelty
[43:20] Oh unmuting would be great
[43:23] sorry about that well for me
[43:26] I guess I wanna understand
[43:28] as far as you si utilizing cori
[43:30] and and and everything you know
[43:32] are you for as far as the IT
[43:33] infrastructure are you using the
[43:35] current IT infrastructure this
[43:36] is or or my understanding this
[43:39] is a third party tool that
[43:43] you're using and if it's you
[43:44] do you know if there
[43:44] if they
[43:48] re using in cloud or or how are
[43:50] they using it IT wise and how
[43:51] involved I really want to know
[43:53] really how involved was our IT
[43:55] security and involved in
[43:57] usingoreAii in in your
[43:58] decision making and and and that
[44:01] director Lindblom
[44:05] thank you for the question so
[44:07] the way the technology works it
[44:09] is it is a third party tool that
[44:12] Optum uses and it uses APIs to
[44:13] to connect with our medicaid
[44:16] management information system so
[44:19] it will make a call for example
[44:21] if a provider puts in a
[44:22] transaction control number for a
[44:23] claim that they want status on
[44:25] it would make a call to our MmIs
[44:27] and then the status of that that
[44:28] claim would be
[44:30] would be returned.
[44:33] in terms of the second part of
[44:34] the question is that something
[44:35] that
[44:37] you might be able to to weigh
[44:41] in on IT security I think
[44:43] andrep Kelty would you like to
[44:45] just restate your questions so I
[44:46] don't add my own
[44:49] interpretation so that our IT
[44:50] folks can answer that.
[44:53] Sure I wanted to know how
[44:54] involved they were in the
[44:56] selection of Kitoreiii and the
[44:58] and the safety of it with the
[44:59] information that's gonna be
[45:00] utilized in or being sent and
[45:02] driven into it I cause I know
[45:05] that they're not a US company so
[45:07] that bothers me a little bit
[45:08] but I just wanna know how
[45:10] closely our own IT people were
[45:12] involved in choosing it and and
[45:14] and how do they feel with as far
[45:16] as the security of that that
[45:16] it's not a US company
[45:21] thanks repkelty thank you Rep
[45:22] Kelty
[45:26] so OIT was involved in the
[45:27] approve the review and approval
[45:29] process. so they were in
[45:31] there the selection was actually
[45:32] through our contractor our
[45:33] contractor chose their
[45:35] subcontractor submitted that to
[45:37] us to request to be able to use
[45:39] this. We then ran it through the
[45:42] process with OIT to get OIT to
[45:45] review and approve it from
[45:46] their through their Ai process
[45:47] as well as through the security
[45:51] and privacy aspects of
[45:52] their standard review process.
[45:52] so
[45:53] it followed their standard
[45:54] review process.
[45:57] that's helpful and did you say
[45:58] that the contractor is Optum
[46:02] provider call center contractors
[46:06] optimum yes. soopptum chose core
[46:07] Ai OK.
[46:10] mm hm
[46:11] yeah
[46:12] OK
[46:13] thank you
[46:20] very good. let's move to
[46:23] Olivia and we we took oh gosh
[46:24] I'm so bad rett Pascal keep
[46:25] going.
[46:29] so repelty I'm interested to
[46:30] know what country
[46:32] uhoreaii is from.
[46:38] repelty do you know
[46:40] I believe it's from India.
[46:43] did she say India?
[46:47] good yeah India that we're
[46:48] getting some nods here at the
[46:49] table.
[46:51] OK.
[46:55] OK. thank you and thank you
[46:59] go ahead please thank you good
[47:01] morning committee. My name is
[47:02] Ashleyennzo and I'm the director
[47:03] of fraud waste and abuse at
[47:04] Colorado Medicaid.
[47:06] I'm here to talk to you guys
[47:08] about hickpuff's new analytic
[47:09] tool that we're utilizing to
[47:11] identify and detect fraud waste
[47:13] and abuse patterns in claims and
[47:14] billing information that's
[47:14] submitted by providers.
[47:17] hickpuff has contracted with
[47:19] Oliviaanalytics through standard
[47:21] procurement efforts to acquire
[47:23] licenses to their trademarked
[47:26] FWA finder FWA being fraud waste
[47:26] and abuse.
[47:29] FWAiner is a service as a
[47:31] software or aA platform that
[47:33] enables state medicaid agencies
[47:35] to identify and investigate
[47:36] providers for suspected program
[47:37] integrity concerns
[47:41] the tool serves as a deployment
[47:42] pipeline and reporting platform
[47:44] for statistical models that are
[47:46] developed by internal staff as
[47:46] well as allevia staff.
[47:49] those statistical models are
[47:51] used to assign risk and generate
[47:53] leads for potential fraud
[47:55] investigation and to identify
[47:56] where there are anomalies in
[47:59] billing among peer groups. The
[48:01] tool only runs algorithms that
[48:03] are developed by human analytics
[48:04] staff and are in no way
[48:05] developed by generative
[48:07] artificial intelligence there
[48:09] are nollMs this is solely our
[48:11] team developing these models
[48:13] within the tool or the Olivia
[48:14] staff putting those models
[48:14] within the tool.
[48:18] the platform has over 150
[48:19] analytic models and hick puff
[48:21] analytics staff have verified
[48:23] approximately75 of those models
[48:25] as accurate to run against our
[48:26] medicaid data.
[48:29] to give you a sense of kind of
[48:30] the analytic models that we are
[48:31] running we have a model
[48:33] that's for impossible hours
[48:35] which examines providers billing
[48:36] more than 24 hours in a day.
[48:39] we have a utilization outlier
[48:41] model which examines per member
[48:42] per month paid amounts to
[48:43] providers that measures them
[48:45] against their peers we have
[48:47] spike models that detect
[48:49] providers with large sudden
[48:50] increases in their billing
[48:53] and then the network analysis
[48:55] model which flags providers who
[48:56] may have connections to known
[48:56] risky providers.
[49:00] the tool stores the analytic
[49:02] models and runs them monthly to
[49:04] generate the leads that we then
[49:06] look at or the focal points.
[49:07] So do you want to make it
[49:09] crystal clear that the tool
[49:11] takes absolutely no action
[49:12] whatsoever. These leads come out
[49:15] in a statistical mathematical
[49:17] sense we then have our data
[49:18] analysts as well as our
[49:19] investigation and case managers
[49:21] who take a look at those leads
[49:23] to then determine if there's
[49:24] some additional action that
[49:24] needs to be taken
[49:27] Humans manually review all of
[49:29] the data and the leads that come
[49:30] out of the tool they manually
[49:32] open cases for further
[49:33] investigation we manually
[49:35] collect additional documentation
[49:39] and we manually initiate adom's
[49:41] audits and so forth. The
[49:43] platform runs on Olivia's Aws
[49:45] cloud environment which is soC2
[49:47] type 2 and high trusts certified
[49:48] and includes all national
[49:49] institute of Standards and
[49:52] Technology controls the tool has
[49:53] received OIT approval for use
[49:57] after reviewing the security
[49:58] and data pros protocols prior to
[49:59] go live. There's also a
[50:01] multifactor authentication for
[50:02] the users to log in.
[50:05] The tool connects to Hickpuff's
[50:07] data warehouse which has also
[50:09] been approved by OIT for proper
[50:10] security measures and controls
[50:13] and then the FWAiner receives
[50:15] monthly copies of flat files of
[50:16] data from the warehouse that are
[50:17] used in the models
[50:20] so just to go over real quick
[50:23] who utilizes the tool. The tool
[50:25] is only used by fraud waste and
[50:27] abuse analytics staff. it is not
[50:28] used by members there is no
[50:29] logins for providers other
[50:31] agencies have no access to the
[50:33] tool in fact a lot of our
[50:35] other offices do not have access
[50:36] to the tool it's solely for our
[50:37] division.
[50:39] the tools not used by any
[50:41] external parties and it does not
[50:43] connect to the internet other
[50:44] than the cloudbased platform
[50:45] where we log in so there's again
[50:48] no no large language models
[50:49] there's no generative Ai nothing
[50:50] that is used within the tool.
[50:53] and just a little bit as to
[50:55] why we're using the tool we have
[50:57] statistical analysts that have
[50:58] been creating models like this
[51:01] for decades at heckpuff. The
[51:03] problem is that the tool we
[51:04] didn't have a tool that could
[51:05] store all of these models so
[51:06] they would create things in R or
[51:09] in python or in sequel be able
[51:11] to manually run those models on
[51:13] occasion or on a regular basis
[51:14] and it would be very difficult
[51:16] to cross compare models. The
[51:17] tool allows us to store
[51:18] everything within one place so
[51:20] it's automatically running on a
[51:20] monthly basis. we can still
[51:24] code within the tool we can code
[51:25] externally from the tool and
[51:27] deploy these models within the
[51:29] tool or we can use predefined
[51:31] models that have come from
[51:32] analysts at the Olivia
[51:35] company
[51:39] we are utilizing this tool to
[51:41] again just help to identify more
[51:43] and more areas where we may
[51:44] need to to put in some
[51:45] additional effort. Antoinette
[51:47] said that she's responsible for
[51:49] saving time. I'm responsible for
[51:51] finding money. and so this
[51:53] tool is helping us to really
[51:54] actually hone in on the bad
[51:55] actors that we may need to look
[51:56] at which is actually
[51:57] advantageous for our providers
[51:59] because we no longer have to
[52:00] look at the group of the
[52:01] providers as a whole to say
[52:03] who's doing what bad y'all are
[52:05] in trouble. now we can say these
[52:06] guys look a little bit different
[52:07] than the
[52:07] rest of your peer group and so
[52:09] we're gonna pull them out and do
[52:10] a little bit more investigation
[52:11] without impacting other
[52:12] operations.
[52:19] I will pause there you ok we are
[52:21] going to now move to quick
[52:24] questions on this and then we're
[52:25] gonna welcome the CBMS crew to
[52:28] come up I do seehett Pascal
[52:29] has her hand up but I saw rep to
[52:32] tone first so we'll do rep to
[52:33] tone rep pascal and then
[52:36] senatorball rep to tone vice
[52:36] chair to tone thank you madam
[52:37] chair
[52:38] thank you madam chairir.
[52:42] compared to the models that you
[52:44] have been previously running
[52:47] individually how does the
[52:47] accuracy compare to this model
[52:54] and have you noticed any marked
[52:55] increase in savings of time
[52:59] using this model compared to the
[53:01] old model because if you are
[53:02] having more you know less
[53:03] accuracy with this model and you
[53:06] have to follow back up or follow
[53:09] leads that are no good then you
[53:11] may not be saving time so just
[53:12] curious about
[53:15] that.rectordurienzo thank you
[53:17] for the question. so as in
[53:19] response to the accuracy of the
[53:20] models I don't have metrics
[53:20] because again these are
[53:21] analytics staff that are still
[53:23] coding within the tools so their
[53:25] models are the ones that are
[53:26] being deployed within the tool
[53:29] unless we're using a predefined
[53:30] model which they're also
[53:32] verifying as accurate. So I
[53:33] think that the predefined models
[53:35] as well as the ability to code
[53:36] within the tools save
[53:37] administrative time from us
[53:39] having to use everything
[53:40] external to the tool and then
[53:40] deploy it
[53:43] but the the tool is
[53:45] definitely allowed us to look
[53:47] at things a lot quicker and on
[53:49] slide two you'll actually see a
[53:50] little bit of what we've been
[53:54] looking at within the metrics
[53:56] so we've helped to generate82
[53:57] new leads with the tool for
[53:59] review and prioritization of
[54:00] actions and just to give you
[54:01] guys a little bit of a a
[54:03] breakdown as to what we're
[54:05] seeing so in particular
[54:06] groups we're actually utilizing
[54:08] the tool to identify prepayment
[54:10] efforts where providers could be
[54:10] placed on prepayment
[54:13] to monitor their billing and
[54:15] help correct any actions we
[54:17] have4 providers of a certain
[54:18] provider type that are on
[54:19] prepayment review right now that
[54:20] we've already generated about
[54:23] $200,000 in savings in six weeks
[54:25] just by having those very
[54:27] specific providers on prepayment
[54:29] review in some other areas we're
[54:30] utilizing the tool to look at
[54:32] post p analytics to determine
[54:34] who we may need to audit next.
[54:36] We right now have a risk pool of
[54:38] approximately 12 providers in a
[54:40] couple provider types that is
[54:40] around
[54:41] I can give you the exact number
[54:43] we'll say tens of millions of
[54:44] dollars wow
[54:47] nice work rett Paskell and then
[54:48] Senator Ball.
[54:51] thank you madam chairir. I was
[54:53] just curious for each of these
[54:54] three tools how long have they
[54:54] been deployed?
[54:57] I'll go first since my mic's
[55:01] onsio thank you the tool went
[55:02] live September of last year for
[55:05] the livia FWAiner and we
[55:06] received CMS certification in
[55:06] May of this year.
[55:13] What about core Aii core Ai was
[55:14] implemented in May 2025.
[55:16] and then what about the member
[55:20] three months 3 months ago.ice.
[55:22] did you get all that
[55:24] Very good
[55:25] Senator Ball
[55:31] thank you madam chairir. for
[55:34] Olivia of the82 new leads how
[55:37] many of those were actually
[55:38] fraud waste and abuse
[55:42] director Durizio. I would have
[55:44] to get the the breakdown. I mean
[55:45] they're they're all classified
[55:46] as potential fraud waste and
[55:48] abuse.raud comes with an element
[55:51] of criminal intent and so those
[55:52] cases are typically less than
[55:55] the waste and the abuse cases
[55:56] we can't comment on cases
[55:58] that may have been referred for
[55:59] investigation because they may
[56:00] be with law enforcement so we
[56:01] don't want to impede their their
[56:04] work but they're all considered
[56:06] fraud waste and abuse potential
[56:06] issues
[56:08] very good
[56:13] thank you so much. we
[56:15] appreciate you all. very
[56:17] good. we're gonna move on and
[56:20] get CBmS up, ir say if I may I
[56:23] did phone a friend core Ai is
[56:25] based in California. Oh OKok,
[56:26] very good. thank you.
[56:31] pascal yeah
[56:34] might have used some Ai to try
[56:37] to see what the story was there
[56:41] but yeah it's yeah it's it's it
[56:43] seems like it has a significant
[56:46] presence in India
[56:48] includingiyderabad. I always
[56:48] have trouble with that name
[56:52] but yeah it looks like it's
[56:53] not actually established there
[56:56] I have
[56:57] I have a question
[57:00] here we go very good. Miss mac
[57:21] and the floor is yours whenever
[57:22] you are ready.
[57:24] you so much madam Chair.
[57:29] right my name is Nina Mac. I'm
[57:30] the chief product officer for
[57:31] Kobe's the Colorado
[57:32] benefitnefits eligibility and
[57:34] enrollment systemstems which
[57:37] includes CBMS and peak and
[57:38] I'm here to talk about the two
[57:42] Ai uses in CBmS the first is
[57:44] the use of hyperscience Icr Ocr
[57:45] so that's intelligent character
[57:47] recognition and optical
[57:50] character recognition this is
[57:51] used to help with document
[57:54] processing. so right now when
[57:55] folks have to apply report a
[57:56] change
[57:59] or renew their benefits they
[58:00] often need to send in paperwork
[58:01] they can do so digitally but
[58:02] several people still choose to
[58:05] send in paperwork through the
[58:07] mail in paper format and when
[58:09] that happens then caseworkers
[58:10] need to process that
[58:11] documentation manually enter the
[58:13] data into CBmS and then that can
[58:15] obviously be time consuming and
[58:16] then you run the risk of fat
[58:18] fingering and making mistakes
[58:19] when you're transcribing from a
[58:21] document into into the system so
[58:24] hypersciences ICr Ocr solution
[58:25] extracts the data from the
[58:26] documents the
[58:29] documents get scanned in then it
[58:30] gets sent to hyperscience Icr
[58:32] where it will extract the data
[58:35] convert handwriting into text
[58:36] and then upload that data into
[58:39] CBMS the accuracy of that
[58:41] process is over 99.8% accurate.
[58:43] it is very good at when it
[58:45] recognizes text, it recognizes
[58:47] it accurately. and we are
[58:50] using this currently for about8
[58:50] types of documents that can be
[58:51] submitted
[58:53] next slide
[58:59] and all anytime data is
[59:01] extracted it is available in
[59:03] CBMS screens so that workers can
[59:04] review it they can see which
[59:06] fields were changed by ICr they
[59:08] can correct errors they can also
[59:10] entirely revert the ICr data
[59:12] changes back to the prior
[59:13] state if they see any issues
[59:14] with the data entry
[59:18] if there is handwritten data and
[59:20] hyperscience rcr it does not
[59:21] cross the threshold for
[59:22] confidence that it can read it
[59:23] correctly it will flag it for
[59:25] human review rather than trying
[59:26] to attempt to extract that data
[59:27] and upload it to CBmS.
[59:31] and there is the ability for
[59:33] humans to be able to say whether
[59:35] or not it was accurate in and
[59:37] that and that helps it train and
[59:38] get better in recognizing the
[59:39] text over time
[59:43] so in the past 12 months
[59:45] over 200,000 documents have gone
[59:47] through ICr so that workers
[59:48] did not need to complete the
[59:50] manual data entry for those
[59:52] and a couple of the key things
[59:53] besides the time savings with
[59:55] that is also some of the
[59:57] impacts for for for the clients
[1:00:00] that we're trying to serve. So
[1:00:00] for example when a
[1:00:02] redetermination or renewal for
[1:00:02] medicaid comes in
[1:00:06] it can it will check that
[1:00:08] there's a signature on that page
[1:00:09] and as long as there's a
[1:00:11] signature it will mark in CBmS
[1:00:13] that the redetermination has
[1:00:15] been received which will then
[1:00:16] start that process so that the
[1:00:17] case cannot be closed for lack
[1:00:19] of returning the paperwork
[1:00:21] otherwise if you had to wait
[1:00:23] for a person to enter in that
[1:00:24] data then there is a period
[1:00:25] between when the document was
[1:00:26] submitted and when somebody
[1:00:27] processes it where there could
[1:00:29] be you know a chance for then
[1:00:31] that case to get closed instead
[1:00:32] of it automatically being marked
[1:00:33] so if the person did what they
[1:00:33] needed to
[1:00:35] do then their case can't be
[1:00:37] affected and then similarly
[1:00:39] for snap if a client reports
[1:00:41] that they have no changes then
[1:00:42] their benefits can automatically
[1:00:43] continue rather than wait for a
[1:00:44] worker to process that
[1:00:44] paperwork.
[1:00:47] I'll pause there before the
[1:00:49] second use case. Oh I appreciate
[1:00:51] that. How did you know? I'm
[1:00:53] excited that that will hopefully
[1:00:55] that redetermination renewal
[1:00:57] as well as the snap is just
[1:00:59] automatic. I hope that really
[1:01:01] helps people stick to their
[1:01:02] benefits that's a very
[1:01:04] compassionate IT use I would say
[1:01:04] repelty
[1:01:10] I was gonna mention something
[1:01:12] about the the India
[1:01:13] connection withoreiii.
[1:01:15] that
[1:01:18] I do that now or at the end and
[1:01:21] let's do it. let's just do it
[1:01:23] now. we're good. and then you
[1:01:25] come right back just hold that
[1:01:27] thought Miss mac OK thank you I
[1:01:28] just I just want to pick
[1:01:29] something up although coreyii
[1:01:31] has the headquarters in Orlando
[1:01:35] how in Florida but the research
[1:01:37] and the development which is you
[1:01:39] know when we're working with
[1:01:41] government entities and that the
[1:01:42] the security of where the actual
[1:01:43] data is coming out of I don't
[1:01:44] care what they have on officeice
[1:01:45] or they
[1:01:47] print their paper whatever that
[1:01:48] is I care about where the where
[1:01:49] the research and development
[1:01:51] where were the core of the the
[1:01:53] crux of the software's coming
[1:01:55] from and that is out of India. I
[1:01:56] just want to put that out there
[1:01:57] because I don't want us to think
[1:02:00] this is a US they have a base in
[1:02:03] in in in in in the United States
[1:02:04] but the but the research and
[1:02:07] development of of the actual
[1:02:09] software comes out of India so I
[1:02:11] just want to make that clear to
[1:02:12] everyone when they're going
[1:02:13] forward and and looking at the
[1:02:14] security and where our data you
[1:02:14] know
[1:02:16] possibly could be going and
[1:02:18] they're expanding that area of
[1:02:20] their company in India. I just
[1:02:20] want to put that out there
[1:02:23] yeah you know and I appreciate
[1:02:25] it and I would say that data
[1:02:26] ownership
[1:02:29] is a concern all around and I
[1:02:31] think that what your statement
[1:02:34] is is a reflection of that as
[1:02:36] well. So I do appreciate you
[1:02:37] mentioning that. I do see the
[1:02:39] director of hip puff nodding his
[1:02:43] head so noted thank you rep
[1:02:44] Kelty thank you thank you
[1:02:46] Miss Mac.
[1:02:53] Thank you madam Chair. The
[1:02:55] second use case is a tool that's
[1:02:57] called panagrame Are Natural
[1:02:59] Dialogue Assistant. this is
[1:03:01] currently state facing work this
[1:03:03] is only is being used by state
[1:03:05] workers currently so when
[1:03:08] somebody has a question about
[1:03:10] policy wants to know how they
[1:03:12] should if they have a question
[1:03:13] they want to know a citation
[1:03:15] they can use this panda chatbot
[1:03:17] instead of needing to look
[1:03:19] through hundreds thousands of
[1:03:20] pages of documents and try to
[1:03:20] sift through and find the
[1:03:21] appropriate answer
[1:03:24] it provides answers in plain
[1:03:26] text along with the citation
[1:03:29] to where it is and it is
[1:03:29] specifically trained on
[1:03:31] documents that are uploaded by
[1:03:34] CdHS programme area subject
[1:03:35] matter experts so a specific set
[1:03:37] of documents the question can
[1:03:39] be typed in they get the answer,
[1:03:40] they get the citation
[1:03:43] the
[1:03:45] this is currently in a pilot
[1:03:47] phase so as I mentioned only a
[1:03:49] handful of state program staff
[1:03:50] have access to this this is so
[1:03:51] that they can test the kind of
[1:03:53] answers that come back before
[1:03:55] releasing it to county workers
[1:03:57] again they're given those
[1:03:57] references so that they can
[1:03:59] verify the accuracy. there's
[1:04:01] also a feedback mechanism for
[1:04:02] them to report if the response
[1:04:05] was unsatisfactory and they
[1:04:07] can only access those documents
[1:04:08] that have been specifically
[1:04:10] approved by CDHS for it to use
[1:04:10] and so if you ask a
[1:04:14] question like what is your
[1:04:15] favorite ice cream? it will not
[1:04:17] return an answer. it will say I
[1:04:18] cannot provide that I cannot
[1:04:19] answer that question. so it is
[1:04:21] very limited in terms of the
[1:04:23] questions it is willing and able
[1:04:25] to answer and currently that
[1:04:26] testing in the pilot phase has
[1:04:27] been rather limited just
[1:04:29] because there are so many other
[1:04:30] priorities when this was first
[1:04:31] implemented a whole lot of
[1:04:33] things happened right afterwards
[1:04:35] and so making this be
[1:04:36] exactly what it needs to be
[1:04:37] before we turn it on for other
[1:04:38] workers hasn't been the first
[1:04:40] priority and now as we're
[1:04:40] looking at replacing CBmS we are
[1:04:41] looking at
[1:04:43] implementing this a bit more
[1:04:45] holistically. and looking at
[1:04:46] making sure that we have the
[1:04:47] right tools and and the right
[1:04:49] use cases because it's not just
[1:04:50] the county workers who need this
[1:04:51] but potentially call center
[1:04:53] agents and other folks and so we
[1:04:54] want to make sure we have the
[1:04:55] right tool in place to support
[1:04:56] all folks and that we have the
[1:04:58] right plan in place to pilot and
[1:04:59] get the the testing data that we
[1:05:00] need before rolling it out more
[1:05:01] widely.
[1:05:03] and really nice work on that
[1:05:07] acronym too it's really cute but
[1:05:10] oh gosh I would say this is
[1:05:14] this is so cool and I really
[1:05:17] see this as something that
[1:05:18] would be really helpful for
[1:05:21] every agency I bet and so I'm
[1:05:23] wondering if we can't do this
[1:05:25] really well in hick puff and
[1:05:29] then or CDHS both that's right
[1:05:30] it's a combined and
[1:05:33] and then see if we can't
[1:05:35] implement this elsewhere. I mean
[1:05:37] gosh legislators would even
[1:05:39] enjoy something like that but ok
[1:05:41] I see senator Ball has his
[1:05:43] hand up. I think no I'm sorry
[1:05:45] rep Kelty I can't see things
[1:05:46] apologiesrep Kelty
[1:05:49] oh no I just forgot to to take
[1:05:51] my hand out it's an old hand
[1:05:55] it's an old hand OKok no worries
[1:05:56] no worries
[1:06:01] yeah rett escal. thank you madam
[1:06:02] chairir. this is I guess more
[1:06:04] of a process question for the
[1:06:07] board committee it I am
[1:06:09] very interested in finding out
[1:06:12] the readiness ofcBMS for work
[1:06:14] requirements starting in
[1:06:18] January2027 and I'm wondering do
[1:06:19] we have that on the agenda
[1:06:20] somewhere in the future
[1:06:25] Miss mac do you want I know we
[1:06:27] don't have it on the agenda yet
[1:06:29] but I know it will be coming. so
[1:06:31] do you have any idea when we
[1:06:34] will be doing that is there I
[1:06:36] guess the way this usually goes
[1:06:38] please director make yourself
[1:06:41] make your way to the table.
[1:06:43] usually we get our IT requests
[1:06:45] budget requests if it's
[1:06:47] under500,000 it wouldn't come to
[1:06:49] us I'm thinking I don't know if
[1:06:50] the work requirements are going
[1:06:52] to require an IT project but
[1:06:57] yeah so at some point it would
[1:06:59] be yeah it would be good to get
[1:07:01] an update on that project even
[1:07:03] if it's not a new or a
[1:07:06] continuing appropriated ask. so
[1:07:07] chair marksman I would tell you
[1:07:09] that this is already scheduled
[1:07:10] in the next meeting or two at
[1:07:12] the Medicaid Commission. So
[1:07:13] there'll be an overview of
[1:07:15] what's going on with work
[1:07:16] requirements there certainly
[1:07:17] that may likely generate
[1:07:21] inquiries from the team here.
[1:07:22] this is applying mostly existing
[1:07:26] technologies so it's more about
[1:07:27] them looking at the policy and
[1:07:28] the implementation
[1:07:31] as I had opportunity to talk to
[1:07:32] the commission
[1:07:35] what was the commission lastst
[1:07:36] week
[1:07:37] you know one of the things that
[1:07:38] we're making sure that we're
[1:07:39] doing is we're continuing to
[1:07:40] keep that North Star up how do
[1:07:41] we keep everyone
[1:07:43] is entitled to be covered
[1:07:45] covered starting with reminding
[1:07:47] people that there is an appeals
[1:07:48] process that retains their
[1:07:50] benefits during any time where
[1:07:51] they believe that they have not
[1:07:51] been
[1:07:53] treated appropriately by changes
[1:07:55] in process changes in technology
[1:07:57] or other things like that so
[1:07:58] we're we're making sure that
[1:07:59] we're very clear in that kind of
[1:08:01] safety net that says we can't
[1:08:02] just take it away from you and
[1:08:05] there you are. OK. that sounds
[1:08:09] good. while you're here mean
[1:08:11] I so and I'm also assuming we
[1:08:12] will get an update with CBms
[1:08:13] just our quarterly update we
[1:08:14] usually get that so
[1:08:17] Miss Mac will you be bringing
[1:08:19] information to us at our
[1:08:20] quarterly update.
[1:08:23] thank you madam chairir. we are
[1:08:24] always happy to update you
[1:08:26] whenever you want. we have
[1:08:27] submitted both in writing and
[1:08:29] then when you ask happy to come
[1:08:31] here before you and speak to it
[1:08:33] as well. OK, very good. we're
[1:08:35] trying to get to a model where
[1:08:37] we're doing more face to face so
[1:08:39] that
[1:08:43] we get so many emails and things
[1:08:45] to read that often we don't get
[1:08:46] to it but if you're sitting
[1:08:49] right in front of us we do so
[1:08:50] thank you for that. We will
[1:08:50] likely be adding
[1:08:53] you to an upcoming quarterly
[1:08:54] update on the CBmS to get that
[1:08:55] question answered
[1:08:59] I was gonna ask you do for
[1:09:01] all of the systems we heard
[1:09:06] about today are our clients are
[1:09:08] members are our folks who are
[1:09:10] using these systems not the ones
[1:09:12] that are just internal but
[1:09:15] the others do they know that Ai
[1:09:17] is being used iss there some
[1:09:19] sort of a message so that they
[1:09:21] know that Ai is being used
[1:09:24] core Ai or some other public
[1:09:25] fa
[1:09:26] cing Ai
[1:09:30] so Cha thanks for your thank you
[1:09:31] for your question. my
[1:09:32] recollection is that we have not
[1:09:37] predisclosed to laution here if
[1:09:39] you go beyond here you will be
[1:09:40] interacting with artificial
[1:09:41] intelligence we know that
[1:09:43] there's a number of these that
[1:09:47] are are are general questions
[1:09:48] that are being managed. I know
[1:09:51] with the member direct with the
[1:09:54] member interactions there are
[1:09:55] questions that they are
[1:09:57] specifically told like around
[1:09:59] PhI that they should not put in
[1:10:00] because of those things
[1:10:02] Antoinette you disclosure on
[1:10:02] yours
[1:10:05] toronto yes thank you madam
[1:10:09] chairir yes every time they
[1:10:11] open up the chatbot we remind
[1:10:13] them so if they open and close
[1:10:15] it and open it again we remind
[1:10:17] them that they are using Ai and
[1:10:21] they consent to it and we
[1:10:23] also remind them to not put
[1:10:25] protected personal information
[1:10:26] in it.
[1:10:33] very good. right. oh yes one
[1:10:34] other piece I did not share
[1:10:37] earlier we did some extensive
[1:10:41] member testing with probably
[1:10:44] 20 to 30 members prior to launch
[1:10:44] as well.
[1:10:49] on der ful Well good luck and we
[1:10:51] will see you for our next update
[1:10:52] and we'll just be in touch when
[1:10:54] that will be so very good. thank
[1:10:55] you
[1:11:09] do that. OK, I would like to
[1:11:12] welcome our next step which is
[1:11:13] CDLE
[1:11:50] OK.
[1:11:53] I believe that we have deputy
[1:11:55] executive director and chief
[1:11:59] communications officer Havend
[1:12:02] and then we have COO Helen
[1:12:03] Thurman.
[1:12:05] Wonder who is going to kick us
[1:12:08] off? thank you madam chairir.
[1:12:09] I can get us started this
[1:12:11] morning again Chehavin deputy
[1:12:13] executive director for the
[1:12:14] department of laborbor and
[1:12:16] Employment. today we will be
[1:12:17] presenting our experience
[1:12:21] utilizing Ai with some of our
[1:12:24] external applications in the
[1:12:26] really in the job seeker and
[1:12:27] unemployment insurance claimant
[1:12:30] space but we also do have an
[1:12:33] update on a new project Cocomp
[1:12:35] which is replacing the division
[1:12:38] of workers' compensations legacy
[1:12:39] system and moving it into a
[1:12:44] modernized environment and
[1:12:45] so with that I am going to turn
[1:12:47] it over to Miss Thurmond to talk
[1:12:48] about familymi shield.
[1:12:51] very good M Thurman
[1:12:53] you need to hit that button and
[1:12:55] it should turnres up better
[1:12:56] that's it you
[1:12:59] my name is Helen Thurman. I'm
[1:13:00] the chief operations officer. I
[1:13:02] am also responsible for the
[1:13:03] technology strategy and
[1:13:05] innovation for the department.
[1:13:07] I'd like to start off today with
[1:13:10] the family shield so this is a
[1:13:11] product that we put in place
[1:13:14] back in 2024 when the family
[1:13:17] division was first getting going
[1:13:20] and we are receiving our claims
[1:13:21] and and
[1:13:23] employees are seeking their
[1:13:25] benefits for the first time and
[1:13:28] so we needed a way to detect
[1:13:32] fraudulent claims. so this
[1:13:35] particular product today has
[1:13:38] saved our claims investigators
[1:13:41] from an hour of work that they
[1:13:42] would have spent looking at each
[1:13:45] claim documentation down to 1f
[1:13:46] minutes and what the
[1:13:49] what this technology does is the
[1:13:51] AI that is being used here is
[1:13:53] basically looking at the data
[1:13:55] it's classifying the data as
[1:13:57] whether it's an employee seeking
[1:14:01] benefit or an employer data or a
[1:14:03] healthcare provider data so it
[1:14:05] classifies information. it also
[1:14:08] is looking at anomalies. are we
[1:14:11] seeing odd behaviors are we
[1:14:13] seeing duplicate use of emails
[1:14:14] that we have seen before for
[1:14:15] another claim and so this
[1:14:19] is a way for our human
[1:14:21] investigators to save time
[1:14:23] looking at the claims that are
[1:14:24] coming through.
[1:14:27] so at this time this is solely
[1:14:30] used for the detection and it
[1:14:34] just flags suspicious claims and
[1:14:37] a human a an actual investigator
[1:14:39] will look at all the flagged
[1:14:41] records to determine whether it
[1:14:42] truly is suspicious or not
[1:14:46] currently as I mentioned
[1:14:49] we are looking at about 1000
[1:14:51] records a month and we are
[1:14:53] saving from an hour of work per
[1:14:55] record down to 1f minutes.
[1:15:03] Very good. so it's an internal
[1:15:05] tool that you use to kind of
[1:15:06] figure out kind of what we just
[1:15:09] heard the other that CBmS and
[1:15:12] hickpuff are using to find that
[1:15:14] kind of fraud waste abuse issues
[1:15:15] and then a person's gonna look
[1:15:18] at it and take that case and
[1:15:19] follow up. it's not that this
[1:15:20] perfect.
[1:15:21] thank you. Reppascal
[1:15:25] thank you madam chairir. So in
[1:15:27] addition to saving time is there
[1:15:31] any change on like how much
[1:15:32] fraud waste fraud and abuse is
[1:15:35] being found. Has that changed?
[1:15:36] Oops Miss Berman
[1:15:41] thank you chair. madam
[1:15:42] chairirs
[1:15:43] thank you for the question
[1:15:45] Representative Pascal. so you're
[1:15:47] asking if there's something else
[1:15:49] that we we're finding from this
[1:15:50] I'm sorry I didn't quite catch
[1:15:53] the rettpascal thank you madam
[1:15:55] chair. So I'm asking is the
[1:15:57] fraud rate found changing due to
[1:15:58] the use of this tool. so we have
[1:16:01] thank you for the question
[1:16:05] p re s ent ative Pascal. so we
[1:16:07] have found about 95% of the
[1:16:08] fraud that's coming through
[1:16:11] initially we probably have paid
[1:16:14] out some fraud but at this point
[1:16:16] we are 95% confident that we're
[1:16:16] catching the fraud.
[1:16:24] rettpascal she's got another
[1:16:26] question how do you come up with
[1:16:27] that confidence level
[1:16:28] though when we
[1:16:30] yes go ahead M Thman
[1:16:33] thank you madam chairir.
[1:16:35] sopresentative Pascal so
[1:16:36] basically what we do is we
[1:16:39] compare the actual flagged items
[1:16:41] with real fraudulent cases and
[1:16:44] we track the metric of how
[1:16:46] accurate are we? Is it truly a
[1:16:47] fraud case or is it
[1:16:50] something that we was
[1:16:53] detected and erroneously by the
[1:16:54] the
[1:16:54] the algorithm
[1:16:59] OK so 95% of what's being
[1:17:01] flagged looks like it's really
[1:17:02] fraud. Is that where the number
[1:17:03] comes fromermon
[1:17:07] thank you madam Chair. that's
[1:17:08] correct. OK, thank you.
[1:17:11] good questions. Great. thank you
[1:17:12] for that.
[1:17:15] what's next
[1:17:19] So the next product that we are
[1:17:21] going to share with you about is
[1:17:22] an interviewer
[1:17:25] eightfold interviewer that we
[1:17:27] use on the connecting Colorado
[1:17:30] platform you may already know
[1:17:31] that the connecting Colorado is
[1:17:34] our labor exchange platform and
[1:17:37] we rolled that out last year and
[1:17:39] we have seen great success using
[1:17:42] AI in terms of how eightfold the
[1:17:45] vendor is allowing the job
[1:17:49] seeker to find or locate jobs
[1:17:50] that seem to match their
[1:17:52] resumes. They had this other
[1:17:52] technology and interviewer
[1:17:55] basically it's a practice
[1:17:56] platform so
[1:17:59] a job seeker has a choice
[1:18:01] they're they're not required to
[1:18:03] use this but basically this is a
[1:18:05] tool for them to practice
[1:18:07] interviewing based on the job
[1:18:08] that they're looking at, it will
[1:18:13] be rulespecific mock interview
[1:18:16] format where the AI acts as an
[1:18:18] interviewer and will ask
[1:18:19] appropriate questions that
[1:18:23] pertain to that job and so the
[1:18:25] job seeker is able to now
[1:18:27] practice what that interview
[1:18:29] experience would be like and
[1:18:30] given that out in the industry
[1:18:31] more and more companies are
[1:18:35] actually using AI as a tool for
[1:18:37] interviewing process for
[1:18:38] screening process. this is a
[1:18:39] great tool for
[1:18:41] a job seeker to get practice,
[1:18:43] get comfortable in answering
[1:18:46] questions and interacting with a
[1:18:49] tool. So basically it
[1:18:51] summarizes for the interviewer
[1:18:55] how you did and and allows them
[1:18:57] to practice again and again and
[1:18:58] allows them to focus on the
[1:18:59] areas where they were weak in
[1:19:01] responding to specific
[1:19:03] questions. Again this is
[1:19:05] entirely optional the job seeker
[1:19:07] does not have to use this tool
[1:19:08] it is just an option that is
[1:19:12] we are making available to them
[1:19:14] as a pilot. Currently we are
[1:19:16] scheduled to go live this Friday
[1:19:19] and this pilot will run for six
[1:19:23] months or so we will end by June
[1:19:26] of 2027 and up to 10,000
[1:19:28] interviews will be allowed and
[1:19:31] after that point we will collect
[1:19:33] feedback from the users or
[1:19:34] rather we will be collecting
[1:19:35] feedback throughout the whole
[1:19:37] process but at that point we
[1:19:38] will assess all the feedback to
[1:19:39] determine whether this is
[1:19:41] an effective tool whether it's
[1:19:43] something that our job seekers
[1:19:44] are finding beneficial.
[1:19:49] very good. thank you.
[1:19:51] what
[1:19:56] is what is available now to job
[1:20:00] seekers versus this new tool.
[1:20:01] can you tell me what is
[1:20:03] available at this time
[1:20:04] because I know people are
[1:20:07] getting pegged to particular
[1:20:11] jobs but I'm just curious how
[1:20:13] this fits into what's already
[1:20:15] there. if you could explain that
[1:20:16] please. thank you madam Chair.
[1:20:16] so
[1:20:20] basically right now what we have
[1:20:23] is a labor exchange platform
[1:20:25] where a job seeker can look for
[1:20:27] a particular job based on their
[1:20:29] skills and what they're
[1:20:31] interested in and the platform
[1:20:33] will call up the list of jobs
[1:20:35] that are posted available from
[1:20:38] variety of employers. they can
[1:20:40] also go to our workforce centers
[1:20:42] and get actual training if they
[1:20:45] would like to do training
[1:20:46] with a person. so that is an
[1:20:46] available
[1:20:50] option for them but they would
[1:20:51] have to actually call make an
[1:20:55] appointment and interact with
[1:20:57] somebody at an actual workforce
[1:20:59] center. The other things that
[1:21:00] are available are approved
[1:21:03] providers of training they can
[1:21:05] do online training via that
[1:21:07] option so if they have a
[1:21:09] specific topic that they feel
[1:21:10] they would like to upskill on or
[1:21:14] basically refresh their skill
[1:21:15] set they can do that kind of
[1:21:17] training as well online or in
[1:21:19] person depending on what the
[1:21:22] trainer is providing but this is
[1:21:25] entirely new in that you can
[1:21:26] actually do it whenever you want
[1:21:29] at your home and and do it over
[1:21:31] and over again and practice with
[1:21:32] an Ai tool.
[1:21:34] that's fantastic. I am just
[1:21:35] going to put a little pin in
[1:21:37] this and we will have to talk
[1:21:39] about this when CDLE comes
[1:21:40] before us but
[1:21:43] I'm curious about the vetting
[1:21:45] that we do for the for the
[1:21:48] employers who are a part of our
[1:21:49] system. I've gotten a number of
[1:21:51] complaints that CDLE is actually
[1:21:54] pointing people to like the geo
[1:21:57] prison and the ice facility in
[1:22:00] Hudson as well as instead of
[1:22:02] posting local jobs they're
[1:22:04] seeing a lot of like Amazon and
[1:22:06] you know like really big jobs
[1:22:07] that is not really related to
[1:22:09] this. I wanted to put that on
[1:22:10] your radar
[1:22:13] so that when you come back we
[1:22:14] can address that and I'll make a
[1:22:18] note of it as well. but I
[1:22:21] think that exhausts my questions
[1:22:23] about the Ai interviewer do you
[1:22:26] have any OK very good. thank you
[1:22:27] so much. right
[1:22:29] what's next
[1:22:33] madam chairir we're gonna
[1:22:35] move into our unemployment
[1:22:37] insurance virtual agent solution
[1:22:41] so in the division of Ui we
[1:22:43] although we have pretty steady
[1:22:46] continued claims and initial
[1:22:48] claims we always obviously have
[1:22:49] a claimant population who wants
[1:22:51] more detailed information on
[1:22:54] their claim status. a couple
[1:22:57] of years ago we implemented a
[1:23:00] solution that provides a chatbot
[1:23:00] function
[1:23:05] or more recently an
[1:23:06] interactive voice response
[1:23:09] system when authenticated gives
[1:23:11] the claimant more detailed claim
[1:23:13] information like last payment
[1:23:17] date claim status appeals
[1:23:19] issues not the full breadth of
[1:23:21] claim status but more detailed
[1:23:23] than if they were to just
[1:23:25] utilize the chatbot on its own
[1:23:29] and so there's really two
[1:23:30] systems working together
[1:23:30] currently 3
[1:23:35] one is the the chatbot that does
[1:23:36] not provide detailed claim
[1:23:38] information. it's a
[1:23:41] conversational gen ai solution
[1:23:43] the other is the phone system
[1:23:45] that when authenticated does
[1:23:47] provide more detail and that
[1:23:50] detail is delivered via an API
[1:23:54] on the Ui claims application
[1:23:57] within the system. When a
[1:23:59] claimant interacts with
[1:24:00] theivVR system.
[1:24:03] they are asked for their
[1:24:07] claimant ID their date of
[1:24:11] birth and no social security
[1:24:13] number is exchanged it then
[1:24:15] authenticates with again the
[1:24:18] main mauai plus system to
[1:24:21] pull up that detailed claim
[1:24:23] information. Again the chatbot
[1:24:25] via the website does not allow
[1:24:27] that more detailed
[1:24:30] information delivery. so we
[1:24:30] do
[1:24:33] believe that since implementing
[1:24:35] these tools we've been able to
[1:24:37] provide greater selfservice
[1:24:39] to claimants so when they are
[1:24:43] able to authenticate about50% of
[1:24:45] claimants are doing so. so it
[1:24:48] has reduced dependency on our
[1:24:49] actual staff in the call center
[1:24:53] we are receiving on average
[1:24:55] several 1000 calls into the call
[1:24:57] center a day because of staffing
[1:24:58] we're not able to get to those
[1:25:00] and so this has allowed us to
[1:25:00] provide that service
[1:25:03] to those claimants who
[1:25:05] otherwise were not able to speak
[1:25:06] with an agent.
[1:25:10] we plan on and we'll get to
[1:25:12] this in the next slide.
[1:25:15] expanding this you may have
[1:25:16] seen in the materials previously
[1:25:18] submitted that there were three
[1:25:21] vendors we actually have since
[1:25:23] we submitted this documentation
[1:25:25] terminated the relationship
[1:25:27] with that vendor and I'll get
[1:25:29] into some of the challenges that
[1:25:31] were presented and where we hope
[1:25:32] to go from here
[1:25:35] but I'll pause there about
[1:25:36] current state
[1:25:39] Oh doctorctorhaen can you
[1:25:43] clarify I see the unemployment
[1:25:45] insurance has google is a
[1:25:47] developer bought copy and wao
[1:25:51] twillo twillolio is that what
[1:25:52] you're talking about one of
[1:25:53] those is gone. All three of
[1:25:55] those are gone. all three of
[1:25:56] those are gone. OK will you go
[1:25:57] ahead and explain or are you
[1:25:59] getting into it later we'll go
[1:26:01] to repa to get into it now after
[1:26:04] represent let's do thathettscal
[1:26:05] thank you madam chairir. I just
[1:26:06] wanted to follow
[1:26:09] up on what you said about this
[1:26:11] getting deeper information on
[1:26:13] the claim is a phone system is
[1:26:14] that like a you know push eight
[1:26:17] to find out this push 9 that
[1:26:18] kind of system madam
[1:26:21] chairirrectorhavan yes it is
[1:26:24] oh it is uhivVR but when
[1:26:25] authenticated with a CId or a
[1:26:28] claimant ID can provide a more
[1:26:28] detailed claim information.
[1:26:31] and then do they have a way to
[1:26:32] exit out of
[1:26:35] that if they need to actually
[1:26:37] talk to a human. Yes madam
[1:26:39] chairir they do and and in
[1:26:42] both the chatbot through the
[1:26:43] website and at the end of
[1:26:45] theivVR they do have an
[1:26:46] opportunity either online to
[1:26:48] thumbs up or thumbs down the
[1:26:51] response was it helpful or not
[1:26:53] and likewise at the end of the
[1:26:55] voice system have the
[1:26:56] opportunity to provide feedback
[1:26:59] or speak to an agent. OK, great.
[1:27:00] Why don't you move into our
[1:27:01] vendor conversation. OK thank
[1:27:02] you madam chairir.
[1:27:05] so they said when we were
[1:27:07] preparing for this presentation
[1:27:10] we were currently engaged
[1:27:13] with a vendor who had multiple
[1:27:16] sub vendors and that
[1:27:19] relationship provided
[1:27:21] while it helped us move toward
[1:27:23] the goal of a more seamless
[1:27:25] system it actually presented
[1:27:27] more challenges and so we
[1:27:29] mutually terminated that
[1:27:30] agreement with the vendor within
[1:27:31] the last several weeks and are
[1:27:32] using that
[1:27:35] experience to take a step back
[1:27:38] and our RFP we do believe
[1:27:41] that a single vendor solution is
[1:27:42] preferred I will just say
[1:27:45] that even in that scenario
[1:27:48] there was no exchange of
[1:27:50] between the three vendors in
[1:27:53] EPIi so all of that again
[1:27:55] through the API interface was
[1:27:57] pulling from the main Ui claims
[1:28:00] system that a claimant interacts
[1:28:01] with that their data is stored
[1:28:02] within but as
[1:28:05] I said we were finding some
[1:28:08] challenges in a more seamless
[1:28:09] integration and development with
[1:28:10] multiple vendors.
[1:28:17] we we still are looking for
[1:28:19] 11 solutions that that does
[1:28:22] integrate the chatbot and the
[1:28:25] phone system so that an agent
[1:28:29] has a kind of an end to end line
[1:28:31] of sight into the claim status,
[1:28:32] the interactions with the
[1:28:33] claimant, the full claim history
[1:28:37] and so that is the the ideal
[1:28:39] next step future state for
[1:28:40] the solution will we will be
[1:28:40] exploring next
[1:28:43] makes sense
[1:28:47] OK, very good. Next
[1:28:54] think it's back to me fors some
[1:28:57] discontinued or yeah so we're
[1:28:58] actually going to talk a little
[1:29:00] bit about workers' comp and the
[1:29:02] first product that was on our
[1:29:05] list was thisscovery navigator
[1:29:09] which we were planning to do and
[1:29:10] discovered in the process that
[1:29:11] it wasn't meeting our
[1:29:13] expectations. so this team the
[1:29:16] dying team it's the division
[1:29:18] ofend medical examination team.
[1:29:20] they normally handle about 3000
[1:29:20] medical pack
[1:29:23] et s a year and handling that
[1:29:27] many documents per case which
[1:29:28] often ends up being hundreds of
[1:29:33] pages of records humans were
[1:29:35] basically having to do duplicate
[1:29:37] the records and it was taking
[1:29:39] too much time so the intent here
[1:29:41] was to use technology to help us
[1:29:43] to classify the documents to
[1:29:45] group them and to duplicate them
[1:29:46] and make sure that they were
[1:29:47] actually a
[1:29:51] complete set of documents per
[1:29:53] case. as I mentioned this
[1:29:55] particular product did not meet
[1:29:57] our expectations and so we have
[1:29:59] decided to end this and we are
[1:30:01] pursuing or looking at a
[1:30:05] different solution and that is
[1:30:06] that next slide
[1:30:09] so we're calling it the medical
[1:30:12] packet consolidation and this is
[1:30:15] something that is currently in
[1:30:17] our on our roadmap but has not
[1:30:19] been scheduled yet because the
[1:30:21] codecom platform, this is a
[1:30:24] modernization of an old legacy
[1:30:25] platform that the workers' comp
[1:30:28] has used and that system just
[1:30:30] went live in July with its first
[1:30:33] release. there are three other
[1:30:34] releases planned core
[1:30:36] functionality that is deemed
[1:30:36] higher prior
[1:30:39] ity for that division and so at
[1:30:42] this time the medical packet
[1:30:42] consolidation
[1:30:46] work is slated for at best end
[1:30:48] of next year if we get to it at
[1:30:51] that point. again it would
[1:30:53] not replace a human actually
[1:30:55] reviewing the packets that have
[1:30:58] been put together by Ai it would
[1:30:59] just look at the flags that are
[1:31:01] potentially duplicates of
[1:31:03] records and so it would save a
[1:31:05] lot of time for the the
[1:31:06] specialists in the dying team.
[1:31:13] are there any questions about
[1:31:14] this particular technology
[1:31:19] I just want to make sure I'm
[1:31:21] looking at the right one. I was
[1:31:22] looking at
[1:31:23] Discoverynavigatorplacement and
[1:31:26] chatbot Cocom on the Ai it's on
[1:31:27] the Ai inventory spreadsheet
[1:31:29] which you may not have in front
[1:31:31] of you. So I I can go ahead and
[1:31:33] start talking about the chapa
[1:31:35] too and maybe we can handle all
[1:31:37] three together that sounds good.
[1:31:38] Great thank you
[1:31:41] madam chairir. so the chatbot is
[1:31:42] wait one second please repasal
[1:31:45] oh I apologize
[1:31:50] yeah who's oh it's rep to tone
[1:31:53] and then repelty rep to tone
[1:31:54] vice chair to tone thank you
[1:31:59] thank you madam chair. So I
[1:32:02] see that your using claud in
[1:32:02] this application
[1:32:07] how do you make sure that you're
[1:32:08] not
[1:32:13] having any of the information be
[1:32:16] used for training or be saved
[1:32:18] in some way that could
[1:32:20] compromise somebody's
[1:32:20] information that could be
[1:32:21] protected
[1:32:25] M Thurmon thank you madam Chair.
[1:32:27] thank you for the question
[1:32:29] Representative Taonene. So
[1:32:31] basically all these documents
[1:32:34] are in a closed environment and
[1:32:35] these are only the documents
[1:32:38] that have been submitted by the
[1:32:40] actual claimants and so there is
[1:32:44] no possibility of injecting any
[1:32:47] kind of vulnerabilities or
[1:32:49] prompts I think maybe
[1:32:50] potentially you're thinking of
[1:32:52] so it's only looking at what
[1:32:56] is already within our database
[1:32:57] of the documents that have been
[1:32:58] submitted
[1:33:03] I shared to
[1:33:05] tone go ahead thank you man and
[1:33:07] I my question really wasn't
[1:33:10] answered what I was asking was
[1:33:10] about
[1:33:16] laude being used as a part of
[1:33:16] that back end
[1:33:19] or oh so that was the other
[1:33:21] one that was on the other side
[1:33:23] that was the others because you
[1:33:24] changed the slide back because
[1:33:25] you were on the other side said
[1:33:26] lauded
[1:33:29] and aws and that was the one I
[1:33:31] was asking the question about. I
[1:33:33] saw that too. where was that
[1:33:35] clad was it's the next slide
[1:33:38] slide
[1:33:40] yeah
[1:33:42] yeah AWsex tract with claude
[1:33:47] access through amazon bedrock
[1:33:49] was the question was asking
[1:33:51] yeah that was what I was asking
[1:33:54] is if it's using laude is it is
[1:33:55] that data being used for
[1:33:57] training for claude and is any
[1:33:59] of the information being
[1:33:59] protected and how do you ensure
[1:34:01] that it's being protected if
[1:34:02] it's personally identified
[1:34:02] information.
[1:34:05] gottcha. I think that I think we
[1:34:07] get that question. Who would
[1:34:08] like to feel that Miss Thurman?
[1:34:11] thank you madam chairir.
[1:34:13] thank you for that question
[1:34:14] Representative Taonene. May I
[1:34:16] get back to you with what our
[1:34:17] current processes for ensuring
[1:34:18] that the
[1:34:22] data is not being used for
[1:34:23] cloud. I'm pretty sure it's not
[1:34:25] but I'd like to come back to you
[1:34:26] with a definitive answer.
[1:34:29] we would appreciate that is
[1:34:31] there any way that you could do
[1:34:33] that within the next like5
[1:34:34] business days. Absolutely,
[1:34:36] yesonder. and if you'll just put
[1:34:39] it in writing to our JTC staff
[1:34:41] JTC staff can send it out to all
[1:34:44] of us if that's OK. awesome.
[1:34:44] thank you so much.
[1:34:49] senator I'm sorry rebelunty
[1:34:53] makeman and and actually
[1:34:56] rep to tone asked the the main
[1:34:57] question that I had for that and
[1:34:59] I I have to apologize I'm I'm
[1:35:02] doing this via my my cellphone
[1:35:05] so I can't really see the the
[1:35:07] room other and it's the size of
[1:35:09] a dime so I I don't know who's
[1:35:10] at the table who's not at the
[1:35:12] table but I did have a question
[1:35:13] about the fraud detection. I
[1:35:14] don't know if that person is
[1:35:15] still there or not. I can't
[1:35:16] tell. So
[1:35:20] but the fraud detection you know
[1:35:20] there was a
[1:35:23] program in here about the fraud
[1:35:25] detection. it was the Ai form
[1:35:27] data the family shield yeah
[1:35:31] we've got Miss Thurman here
[1:35:32] who can respond to that
[1:35:34] OK I probably like cause I can't
[1:35:35] see a thing so I'm kind of going
[1:35:37] blind here butries I I'm very
[1:35:39] much in debt you know the IT and
[1:35:40] thet security and stuff like
[1:35:41] that and that's just been kind
[1:35:43] of rattling around in my brain
[1:35:45] right now but you know as we
[1:35:46] know that the fraud detection
[1:35:49] really is a continuous loop, you
[1:35:50] know what I mean rather than
[1:35:52] like a finished product finished
[1:35:53] long r them so and theyment the
[1:35:55] logarithm that they're utilizing
[1:35:58] and I just I wanna know what
[1:36:01] that what does that looking for
[1:36:02] currently right now and what
[1:36:05] is your, your change rate as far
[1:36:07] as keeping up with the the the
[1:36:09] hackers or the anti detection
[1:36:11] the anti anti detectors you know
[1:36:12] what I mean like the ones who
[1:36:13] are trying to constantly go
[1:36:14] around it. what is the the
[1:36:15] change rate that you have for
[1:36:21] that and because with the
[1:36:22] hackers attackers were you know
[1:36:23] whatever you want to call them
[1:36:24] you know they're we're trying to
[1:36:25] keep them from exploiting the
[1:36:27] data right to to get around
[1:36:29] through you know either
[1:36:31] adversarial adaption or
[1:36:32] otherwise you know they don't
[1:36:33] call it
[1:36:35] what model model model
[1:36:35] invasion I believe model
[1:36:37] invasion attacks so what are we
[1:36:39] doing to prevent that and how
[1:36:41] often is that longer than being
[1:36:44] changed to prevent them from
[1:36:44] getting in and getting the data.
[1:36:48] Miss Thurmon thank you madam
[1:36:49] chairir
[1:36:53] thank youpresentative Pascal
[1:36:57] was it is yes no worries so I
[1:36:59] can tell you right that at this
[1:37:03] point we are using 166 active
[1:37:06] indicators to determine whether
[1:37:09] a claim is fraudulent or not and
[1:37:11] of that a lot of it is
[1:37:13] statistical analysis and we have
[1:37:15] certain ones that we absolutely
[1:37:17] look for like duplicate email
[1:37:19] addresses or whether the bank
[1:37:20] account information has is on a
[1:37:20] sanction
[1:37:24] list for AFAFacC and and we look
[1:37:25] to see whether the employer's
[1:37:28] FEIN number is identical to the
[1:37:29] claimant's social security
[1:37:31] number but your question is how
[1:37:34] do we keep up with the changes
[1:37:35] that are occurring in the
[1:37:37] cybersecurity space and for that
[1:37:40] particular rate of change, I
[1:37:41] would like to get back to you
[1:37:43] with the after consulting
[1:37:45] with the team and provide you
[1:37:46] with a fuller response.
[1:37:49] thank you. I appreciate that and
[1:37:50] could you do that within the
[1:37:50] five days?
[1:37:53] OK, I appreciate you thank
[1:37:55] youonder thank you so much and
[1:37:57] yeah that'll be helpful.
[1:38:01] awesome do you want to get into
[1:38:02] the chatbot?
[1:38:07] so this is again for the coca
[1:38:09] platform and the intent here was
[1:38:12] to actually provide our
[1:38:15] claims and and our health
[1:38:17] providers and employers with
[1:38:19] another option for getting
[1:38:21] answers closer to real time
[1:38:24] versus getting on a phone and
[1:38:25] having to talk with somebody
[1:38:27] although that option is still
[1:38:28] available and and I might add
[1:38:31] that this particular division is
[1:38:32] exceptional when it comes to
[1:38:32] responding to
[1:38:35] their phone calls, they get up
[1:38:37] to about 60 calls a day and most
[1:38:39] of the calls are answered within
[1:38:42] 20 seconds or shorter so it's
[1:38:44] not that the folks who are
[1:38:45] calling in are having to wait a
[1:38:47] long time that is not what we're
[1:38:49] trying to solve for here but we
[1:38:53] did want the actual claimant or
[1:38:54] whether it's a provider or
[1:38:55] employer to be able to look up
[1:38:57] the status of their current
[1:39:00] claim or if they needed to find
[1:39:02] another document a way to get to
[1:39:03] that information faster
[1:39:06] and be able to self serve. so
[1:39:08] this chatbot is currently not
[1:39:10] scheduled. it is something that
[1:39:12] is on our roadmap and we would
[1:39:14] like to have this developed
[1:39:17] after all of our core platform
[1:39:19] functionalities are in place and
[1:39:22] at best we're guessing at this
[1:39:23] time that we may be able to get
[1:39:25] to it by the end of 2027.
[1:39:30] Very good
[1:39:33] what's next
[1:39:38] OK I believe I have the next one
[1:39:39] again. this is the snow
[1:39:43] snowflake cortex so this is
[1:39:45] actually for not just one
[1:39:47] division but we are in the
[1:39:50] process of developing a data
[1:39:54] lake for all of CDE and at some
[1:39:55] point in the future we will have
[1:39:57] most of our critical division's
[1:40:00] data on this platform and the
[1:40:02] intent here is to leverage the
[1:40:04] built in AI capability withinn
[1:40:04] snowflake to do the types
[1:40:07] of analytics that we are
[1:40:09] currently not able to do whether
[1:40:12] it's predictive analytics or
[1:40:15] cost modeling future
[1:40:17] modeling that we're not able to
[1:40:19] do today and especially because
[1:40:23] it requires individuals to have
[1:40:24] certain level of skill whether
[1:40:27] it's python or SQl coding we
[1:40:28] wanted
[1:40:31] the ability for folks who are
[1:40:33] I will use the word junior
[1:40:36] junior data analysts or
[1:40:38] business intelligence analysts
[1:40:40] to be able to leverage a tool
[1:40:41] where they can use plain
[1:40:44] language to ask the questions in
[1:40:46] in plain language and get the
[1:40:48] kind of analytic output from a
[1:40:50] platform. So that's the intent
[1:40:52] and at this time this is not in
[1:40:55] production but we are very
[1:40:56] interested in exploring what it
[1:40:56] can do for the department.
[1:41:00] no ther one of those really
[1:41:02] exciting put all the information
[1:41:04] together and let's see how we
[1:41:05] can't use this so I
[1:41:09] that's how I like to use Ai too
[1:41:12] so I think this is brilliant.
[1:41:13] questions, concerns
[1:41:17] I don't see any
[1:41:22] what's next? I think that's all
[1:41:24] of the systems yeah I was like
[1:41:26] I've I've got them all crossed
[1:41:29] off on my list so wonderful.
[1:41:31] I took note of a couple of
[1:41:33] unanswered questions. I just
[1:41:34] want to say thank you so much
[1:41:37] for being willing to really
[1:41:39] dig in I think sometimes we ask
[1:41:41] a question and we don't always
[1:41:44] get the detail of answer that
[1:41:45] we want and so I just really
[1:41:47] appreciate you digging in
[1:41:49] looking forward to hearing about
[1:41:50] how you're keeping
[1:41:55] up with the cybersecurity and
[1:41:56] pretty much all the systems but
[1:41:58] in the family but that's pretty
[1:41:59] much all the systems is where
[1:42:03] that came from and then also
[1:42:05] looking at how we keep our data
[1:42:08] from training clad or in fact
[1:42:10] any of the vendor models that we
[1:42:13] are sharing with so
[1:42:16] appreciate that
[1:42:20] and with that we will dismiss
[1:42:22] you guys. thank you so much.
[1:42:23] thank you madam Chair you madam
[1:42:25] members of the committee thank
[1:42:26] you thank you for the
[1:42:27] presentation did a really good
[1:42:28] job.
[1:42:33] we have up next Department of
[1:42:37] Public Safety. I don't know that
[1:42:38] I see our colleagues here
[1:42:47] they are up at 11:15 I would
[1:42:49] say I'll give us all 20 minutes
[1:42:52] but I'd prefer for them to come
[1:42:53] here because we're going to have
[1:42:56] two parts of this one part is
[1:42:57] going to be an executive session
[1:43:01] and one part is not and so
[1:43:04] for that I would we'll we'll
[1:43:05] take a five minute break. why
[1:43:07] don't we come back here at
[1:43:11] is my clock wrong? what time
[1:43:12] is your say?
[1:43:17] it says 10:4410:45. All right
[1:43:21] we'll come back at 10:50 and
[1:43:22] maybe folks will be here.
[1:43:22] thanks.
[1:50:09] OK, we'll get started. we
[1:50:13] have CDPS in front of us today
[1:50:14] they have
[1:50:15] two sections that they are
[1:50:18] going to cover today. One will
[1:50:24] be one will be in public
[1:50:27] that'll be about the ALPR as
[1:50:30] well as the 911 Aicoms coach,
[1:50:32] those two things and then we
[1:50:33] will be going into
[1:50:36] executive session
[1:50:41] for the following two issues
[1:50:43] which are the motorolamotorola
[1:50:47] camera upgrades and life raft
[1:50:48] and we'll talk about that as we
[1:50:48] get there
[1:50:52] So yeah
[1:50:58] Oh and we're also I missed that.
[1:51:00] thank you. We're also going to
[1:51:03] publicly discuss case guards
[1:51:05] so we'll talk about acction
[1:51:06] Oh and we're also I missed that.
[1:51:07] thank you. We're also going to
[1:51:07] publicly discuss case guards
[1:51:08] so we'll talk about acinovors
[1:51:09] and caseguard all togetherther
[1:51:11] and then we'll go and new
[1:51:13] executive session and we'll talk
[1:51:15] about the other two components
[1:51:20] so OK, very good. Let's get
[1:51:20] started.
[1:51:22] do you have everyone you need
[1:51:30] are we gonna start with you?
[1:51:35] you're just doing the slides.
[1:51:37] very good. So we have chiefef
[1:51:38] Matthew Packer
[1:51:40] or no
[1:51:43] this is oh we've got aaron
[1:51:45] Coning up first gott you. thank
[1:51:47] you. Come on up in your erran.
[1:51:48] Very good.
[1:51:54] No, it's on the it's on the desk
[1:51:55] I believe or on the yeah
[1:51:59] now you're green. very good.
[1:52:02] thank you madam chair and good
[1:52:03] morning members of the
[1:52:06] jointchnology Committee. my name
[1:52:06] is aaron Coney.
[1:52:09] and I am the quality director
[1:52:11] for CBI forensic Services and
[1:52:14] today I'm here to present on
[1:52:15] Caseguard software which is a
[1:52:16] document redaction application.
[1:52:18] go to the next slide
[1:52:23] first of all I want to stress
[1:52:25] that we have not yet deployed
[1:52:27] this software but are still in
[1:52:28] the process of evaluation
[1:52:32] and to give you some context on
[1:52:33] why we are exploring this
[1:52:35] technology our laboratory runs a
[1:52:37] rigorous quality assurance
[1:52:39] program to identify investigate
[1:52:42] and remediate errors and
[1:52:44] potential nonconformances we
[1:52:45] currently document these
[1:52:48] workflows electronically and in
[1:52:49] order to improve external
[1:52:52] transparency and accountability
[1:52:53] we intend to publish these
[1:52:55] workflows to our public facing
[1:52:56] website
[1:52:59] However these documents contain
[1:53:01] sensitive information such as
[1:53:04] names, dates of birth, agency
[1:53:05] information and casespecific
[1:53:09] data that falls under personally
[1:53:11] identifiable information or PIi
[1:53:12] and sieges protection
[1:53:14] we are currently evaluating
[1:53:15] Caseguard to see if it can help
[1:53:18] make this reduction process more
[1:53:19] efficient and accurate
[1:53:21] Once again it's important to
[1:53:23] note that we are still in the
[1:53:25] evaluation phase and have not
[1:53:26] yet deployed this application to
[1:53:27] date.
[1:53:30] Furthermore, any automated
[1:53:31] process will be followed by a
[1:53:34] comprehensive human review to
[1:53:37] ensure absolute accuracy before
[1:53:38] any information is published.
[1:53:39] excellent
[1:53:43] so what value does this tool
[1:53:46] offer? First it pro it provides
[1:53:49] automated first pass redaction
[1:53:51] capable of instantly recognizing
[1:53:53] over 30 categories of PIi and
[1:53:56] other sensitive information and
[1:53:58] can also recognize complex
[1:54:00] patterns like lab and agency
[1:54:02] case numbers. We think this can
[1:54:03] reduce the risk of human error
[1:54:05] that can happen during manual
[1:54:08] scanning and second we think it
[1:54:10] can offer significant
[1:54:10] operational efficiency
[1:54:13] we estimate that this will save
[1:54:14] our team
[1:54:17] up to about 20 hours of labor
[1:54:19] per month while helping us
[1:54:21] process about75 of these
[1:54:22] documents within that time
[1:54:22] frame.
[1:54:26] Finally I want to emphasize that
[1:54:27] this is a nonconsequential tool
[1:54:31] it is purely for operational
[1:54:33] support. it does not make
[1:54:35] autonomous decisions regarding
[1:54:37] public access nor does it affect
[1:54:38] our core operations
[1:54:41] it will simply help us manage
[1:54:42] our workload more effectively.
[1:54:43] Next slide
[1:54:48] and finally I want to address
[1:54:51] security and managements we will
[1:54:53] have a human in the loop review
[1:54:55] which means that we'll
[1:54:55] have a meaningful human
[1:54:57] verification step that's
[1:55:00] required after every automatic
[1:55:01] reaction to s to ensure accuracy
[1:55:05] from a security standpoint case
[1:55:07] guard is a strong fit because it
[1:55:09] runs fully locally. All
[1:55:11] processing happens on our own
[1:55:14] machines no data is transmitted
[1:55:16] to external servers regarding
[1:55:18] management access is strictly
[1:55:20] restricted to the CBI
[1:55:20] forensicervicesuality unit
[1:55:23] the financial commitment is
[1:55:25] minimal it's about $280 per
[1:55:28] month with no long term contract
[1:55:28] required
[1:55:31] and in terms of data life cycle
[1:55:33] unredacted files remain secure
[1:55:35] and our internal CDpS drives
[1:55:38] while only the fully redacted
[1:55:40] versions are moved to our public
[1:55:40] facing website
[1:55:43] and so in summary we hope to
[1:55:45] utilize caseguard to increase
[1:55:47] operational accuracy and
[1:55:48] efficiency and it will not be
[1:55:50] used to make any independent
[1:55:51] decisions on information
[1:55:52] security.
[1:55:57] very good.
[1:55:59] Repkelty has a question I
[1:56:00] saw.
[1:56:05] thank you madam chair. Yeah umm
[1:56:07] apologize I didn't catch your
[1:56:08] your title but I I do have a
[1:56:09] question about this and I think
[1:56:11] this goes across the board with
[1:56:12] anything that we're using as far
[1:56:15] as applications or any type of
[1:56:17] new IT whatever that we're
[1:56:21] using is coming into into the
[1:56:23] realm I guess you could say. so
[1:56:24] as far as you saying this isn't
[1:56:27] an evaluation period for this
[1:56:29] how can you tell me how involved
[1:56:31] with the evaluation of the
[1:56:32] software is our IT
[1:56:35] security or or IT apps
[1:56:37] department when when you're
[1:56:38] going through this evaluation
[1:56:43] that's my question there you go
[1:56:46] and Rep Kelty this is aaron
[1:56:49] Conan quality director of CBI
[1:56:50] Forensic Services Mroning
[1:56:51] thank you
[1:56:53] Yeah thank youre representative
[1:56:55] for the question. yeah that's
[1:56:57] a great question. so our IT
[1:57:00] department is very involved they
[1:57:02] were involved very closely
[1:57:04] with the selection process of
[1:57:06] this software we evaluated
[1:57:09] several different redaction
[1:57:11] software applications and our IT
[1:57:15] department helps us select this
[1:57:17] particular one and so they have
[1:57:19] been very closely involved with
[1:57:21] the selection process and we
[1:57:25] plan to consult with them during
[1:57:26] the evaluation process as well.
[1:57:29] answer I appreciate it. thank
[1:57:30] you.
[1:57:40] OK you said75 documents per
[1:57:42] month. how much time or money
[1:57:45] did you say was saved? you said
[1:57:46] something but I don't remember
[1:57:47] what you said. Do you remember?
[1:57:50] I think it was let me go back to
[1:57:51] my number it was about 20
[1:57:53] hours of labor per month that we
[1:57:55] estimate so you're correct we
[1:57:58] have we estimate about75 of
[1:58:00] these workflows in electronic
[1:58:04] format each month and so we
[1:58:04] estimate about 20 hours within
[1:58:08] the scope of that month on time
[1:58:08] savings
[1:58:11] OK so kind of a higher level
[1:58:13] question that the people at this
[1:58:14] table may not be able to answer
[1:58:16] but I just say cause I know ohIT
[1:58:18] is in the room but
[1:58:21] this would be a good example if
[1:58:23] we have one reduction software
[1:58:25] that's working really well we
[1:58:27] should probably have one that
[1:58:29] works really well for the state
[1:58:31] of Colorado unless there's some
[1:58:34] compelling reason to have75 for
[1:58:37] each75 departments so I just put
[1:58:39] that on the record because this
[1:58:42] is just an example of common
[1:58:44] government work that you have to
[1:58:47] redact documents and so I would
[1:58:48] be very interested and
[1:58:51] in seeing at a higher level,
[1:58:53] not you guys but at a higher
[1:58:55] level how many reduction
[1:58:57] softwares do we have and I'm
[1:58:59] seeing a thumbs up from the back
[1:59:02] so very good. Thank you. Seeing
[1:59:04] no more questions. we thank
[1:59:07] you for that review of Caguard
[1:59:09] and the redaction. It it'll come
[1:59:12] live at some point what are the
[1:59:14] determinations for how you know
[1:59:16] when you're gonna go live I
[1:59:16] guess is a question
[1:59:22] we we we had a position that we
[1:59:23] were going to bring on board
[1:59:26] that one of the duties of
[1:59:28] that this new position was going
[1:59:30] to be to assist with a manual
[1:59:32] review of the redaction process.
[1:59:34] unfortunately we were not
[1:59:35] successful with bringing that
[1:59:38] person on board and so we have
[1:59:39] identified another individual
[1:59:42] with some capacity to do that
[1:59:43] and and so that really that's
[1:59:45] what we were waiting for
[1:59:46] before we redact and put these
[1:59:49] make these publicly available on
[1:59:51] our website and so I would
[1:59:53] anticipate within the next
[1:59:55] several months that we will
[1:59:59] evaluate this and if successful
[2:00:01] we hope to go live with
[2:00:03] making these publicly available
[2:00:04] hopefully early in 27.
[2:00:09] on der thank you so much for
[2:00:10] your presentation
[2:00:14] todayam you. right now we'll
[2:00:16] hear from Chief Packard chief
[2:00:19] thank you very much madam Chair
[2:00:21] and members of the committee
[2:00:23] it's a pleasure to be with you
[2:00:24] on this wonderful Tuesday
[2:00:26] morning. so we'll we'll go
[2:00:27] ahead and start the presentation
[2:00:29] we're as you outlined we'll
[2:00:31] go over the first two systems
[2:00:33] here and happy to take any
[2:00:35] questions and madam Chair I did
[2:00:36] receive some of the questions
[2:00:38] you sent in advance and I'll
[2:00:39] jump into those two when we're
[2:00:40] talking about Axon
[2:00:42] but I just want to start
[2:00:44] high levell and this really does
[2:00:46] apply to all four of the systems
[2:00:48] that we'll talk about today and
[2:00:49] you'll see the slide on the
[2:00:51] screen but overall and going
[2:00:52] through the decision points that
[2:00:54] the committee provided for us to
[2:00:55] address in the presentation
[2:00:59] first off like we don't have any
[2:01:00] computer systems that are making
[2:01:02] actual decisions for our members
[2:01:05] these systems are providing
[2:01:07] inf are prompting members the
[2:01:08] opportunity to make a decision
[2:01:09] based on the technology they are
[2:01:11] there but any enforcement
[2:01:13] related decision or contact
[2:01:15] decision or any of those the
[2:01:17] systems are presenting the
[2:01:19] opportunity in any again any
[2:01:20] consequential decision is
[2:01:21] actually being made by a human
[2:01:22] that works for the Colra State
[2:01:23] patrol in this context
[2:01:26] all of the data is governed by
[2:01:27] strict cigious compliance
[2:01:28] depending on the system whether
[2:01:31] it's through Cis through OIT
[2:01:33] data protocols depending on
[2:01:35] which system that we're talking
[2:01:37] about and where it lives all of
[2:01:38] that is under strict security
[2:01:40] and standard protocol whether
[2:01:40] it's again through the FBI with
[2:01:44] CI or state OIT compliance as
[2:01:46] I already kind of alluded to
[2:01:47] all of the systems that we'll
[2:01:48] talk about have a human in the
[2:01:51] loop the LPRs is a great
[2:01:52] conversation to talk about and
[2:01:52] how that human engagement
[2:01:55] and when I talk about the com
[2:01:56] training you'll understand that
[2:01:57] it's a supervisor that it's
[2:01:58] engaging in those things but
[2:01:59] there will always be that
[2:02:03] human in the loop. talked a
[2:02:04] little bit about security and
[2:02:05] risk as well about through all
[2:02:08] the on the procurement side and
[2:02:09] the first system that we're
[2:02:10] going to talk about is the coms
[2:02:13] coach and that was a long
[2:02:14] onboarding process in fact the
[2:02:16] training component they'll talk
[2:02:17] about we haven't even officially
[2:02:18] started because of all of the
[2:02:20] testing once the vetting
[2:02:22] happened to to on board the
[2:02:22] system so
[2:02:26] doing a lot of work around
[2:02:28] vetting identifying and then
[2:02:31] mitigating risk we have audit
[2:02:33] processes built in sometimes
[2:02:35] that's internal with supervision
[2:02:36] sometimes it's a position
[2:02:38] sometimes it's groups that we
[2:02:39] work with but there's
[2:02:39] independent audits that are
[2:02:40] involved there as well
[2:02:43] and all of our contracts are
[2:02:44] built in a manner that if we
[2:02:45] need to get out we can get out
[2:02:48] I mean all cards on the table
[2:02:50] technology is kind of this place
[2:02:52] whereas you're in and it
[2:02:53] advances and it turns into
[2:02:54] somewhat of a budget decision
[2:02:56] about whether we stay with a
[2:02:58] particular vendor or system or
[2:02:59] try to restart with somebody
[2:03:01] else but the contracts are built
[2:03:04] in in a manner that that we we
[2:03:05] have an out that gets us if we
[2:03:06] need to go in a different
[2:03:08] direction and then all of the
[2:03:09] data that the patrol is putting
[2:03:10] into these systems
[2:03:13] or takes is owned by us and
[2:03:15] we have if we ever choose to and
[2:03:17] I'm not aware of one right now
[2:03:18] where we're actually sending
[2:03:19] that data out so from a data
[2:03:21] security and ownership and life
[2:03:23] cycle we own the data and then
[2:03:24] have the opportunity to make
[2:03:24] decisions on how long we keep
[2:03:25] that data.
[2:03:29] so first off to get more
[2:03:31] specific we'll talk about the
[2:03:32] comms coach by Govorks and again
[2:03:33] this is a system that was on
[2:03:36] boarded specifically number
[2:03:37] one to help with two things
[2:03:39] train our new communications
[2:03:41] officers our dispatchers and
[2:03:42] also to do this quality
[2:03:44] assurance program and so one of
[2:03:46] the challenges particularly
[2:03:48] in in emergency communications
[2:03:49] is recreating an environment
[2:03:51] that's realistic without
[2:03:54] actually talking to callers and
[2:03:55] troopers on the radio and all of
[2:03:56] those things where life could
[2:03:57] be
[2:03:59] at stake and so this software
[2:04:01] allows us to create a training
[2:04:03] environment that is it's really
[2:04:05] quite remarkable what it can do
[2:04:07] realistic voices, realistic
[2:04:09] scenarios emotion all of those
[2:04:11] things that are in the call that
[2:04:13] happened in the real world. this
[2:04:15] system helps us develop and then
[2:04:17] the Ai the intelligence in it
[2:04:19] can then react to how our
[2:04:20] trainee is is you know
[2:04:21] handling that particular call
[2:04:23] for service so pretty compelling
[2:04:25] stuff with that what what we'll
[2:04:26] be able to do with that again we
[2:04:26] haven't actually turned that on
[2:04:27] for training yet
[2:04:29] but we have turned on the
[2:04:31] quality assurance programme
[2:04:33] and this is a normal it's a
[2:04:35] pretty standard practice in
[2:04:36] emergency communications where
[2:04:37] you go back through and have
[2:04:39] some some sort of audit process
[2:04:41] on how call taking is being
[2:04:43] handled and how dispatch calls
[2:04:45] are being dispatched and just
[2:04:47] to give you an idea and we're
[2:04:49] relatively new into using this
[2:04:51] but normally this is a task
[2:04:52] that was assigned to supervision
[2:04:55] in our com centers and on a good
[2:04:56] month we could get about 250
[2:04:57] reviews done a month.
[2:05:00] the first month we turned this
[2:05:02] on we're able to do 13,000
[2:05:04] reviews and so that is going
[2:05:05] through the roop and and
[2:05:06] thanks to the good
[2:05:08] capptainaththle I just got some
[2:05:09] fresh data this morning that
[2:05:12] our we've had high scores like
[2:05:14] we passed with a 96%rcom
[2:05:15] officers so they're doing really
[2:05:17] really incredible work and as it
[2:05:19] turns out and this seems
[2:05:21] predictable the calls where they
[2:05:22] were struggling more were less
[2:05:24] routine calls that are not
[2:05:25] necessarily in the patrol's
[2:05:26] wheelhouse for example, state
[2:05:26] troopers don't do a lot
[2:05:29] of trespassing calls and so our
[2:05:31] com officers might not be
[2:05:31] involved in those much so the
[2:05:33] questions they're asking how
[2:05:34] they're handling although
[2:05:35] they've been trained or not as
[2:05:37] fresh as it would be for example
[2:05:38] as an injury crash or a hit and
[2:05:39] run or a traffic stop or those
[2:05:40] types of things.
[2:05:44] this is this slide up there
[2:05:45] and I'll just kind of let you
[2:05:46] look at this because it's a
[2:05:47] bunch of the more the technical
[2:05:49] aspects of the systems of how
[2:05:51] it's using but again what I
[2:05:54] really want to stress here is as
[2:05:55] we're using this and as we
[2:05:57] planned it roll this out we can
[2:06:00] develop scenarios we we put that
[2:06:02] into the system and then the
[2:06:03] scoring that happens whether
[2:06:05] that's on a QA or on a training
[2:06:07] scenario the results of that are
[2:06:09] all come back to us so it's not
[2:06:10] like the system is taking what
[2:06:10] we're putting in based on s
[2:06:13] ce na ri o s and then using that
[2:06:15] in other places we do own all of
[2:06:17] that and the Ai that it's using
[2:06:19] is based on internal data that's
[2:06:21] put in there by members of the
[2:06:22] state patrol and our training
[2:06:22] staff
[2:06:26] just a couple of quick notes
[2:06:28] there building off that first
[2:06:29] slide that I talked about number
[2:06:31] one accessibility was as as
[2:06:33] noted with as required by
[2:06:34] House211110.
[2:06:36] and then like the cost that
[2:06:38] we talked about of the we have a
[2:06:39] wonderful relationship with the
[2:06:41] AG's office and one of my
[2:06:43] favorite attorneys on the planet
[2:06:44] is our contract attorney there
[2:06:46] and Laurie does great work for
[2:06:47] us and we give her plenty to
[2:06:49] look at but on the strength of
[2:06:51] our contracts with that
[2:06:53] partnership has been a very
[2:06:55] strong and as noted comorks is a
[2:06:56] software program it's no
[2:06:57] hardware on site for us.
[2:07:01] and then again just kind of a
[2:07:02] summary on this one I think this
[2:07:04] is a real value add if if
[2:07:04] nothing else I mean the the
[2:07:06] efficiency is a huge deal for us
[2:07:08] of course but I think the the
[2:07:10] biggest benefit here is we're
[2:07:12] going to be able to have folks
[2:07:13] that are going to get on live
[2:07:15] call taking and live dispatching
[2:07:16] and having gone through the
[2:07:18] system that transition will be
[2:07:19] much more seamless than it would
[2:07:21] have than it had been before. so
[2:07:22] the first time that they're
[2:07:24] dealing with a life saving you
[2:07:26] know a life at risk won't feel
[2:07:27] like the first time anymore and
[2:07:28] that's a pretty compelling thing
[2:07:29] for
[2:07:29] us.
[2:07:31] maybe I'll stop there for any
[2:07:32] questions on the comors.
[2:07:37] Super cool I went and visited
[2:07:40] a group of doctors last summer
[2:07:43] and they're also using
[2:07:44] technology to train people and
[2:07:45] little did I know that I would
[2:07:47] be getting to be trained in this
[2:07:49] real life scenario of giving
[2:07:53] helping deliver a baby with
[2:07:56] this amazing technology but it
[2:07:59] exists because of the real world
[2:08:00] stuff the crises what's the
[2:08:01] patient saying things you're
[2:08:02] not
[2:08:04] expecting there's someone else
[2:08:06] in the room now how do you deal
[2:08:07] with that person? That is a
[2:08:11] perfect example of how to use Ai
[2:08:13] in my mind and just a really
[2:08:16] great use I wanted to be
[2:08:18] clear on something you said that
[2:08:22] you used to do 250 units per
[2:08:23] month and now you're doing
[2:08:26] 13,000 units. what's that unit?
[2:08:28] Is it a training scenario? Is
[2:08:30] that what that is? Thank you
[2:08:31] madam Chair that's on the
[2:08:32] quality assurance piece and so
[2:08:32] that's
[2:08:35] that's 250 reviews of a call. so
[2:08:37] it's one call is one review and
[2:08:39] now we're able to review a much
[2:08:41] much larger review of thosew or
[2:08:43] sampling of those I should say
[2:08:45] that's great thank you I don't
[2:08:47] see any other questions so we
[2:08:48] can move to the next one
[2:08:51] right thanks and we'll move on
[2:08:53] to the the conversation about
[2:08:55] axon and go ahead and go to the
[2:08:56] next slide for me please. one of
[2:08:57] the things in the the the prompt
[2:09:01] to bring us here todays spoke
[2:09:02] specifically about fleet re
[2:09:04] which is our incar camera system
[2:09:06] and then how ALPr can be an
[2:09:08] incorporated to that but I did
[2:09:09] want to like talk a little bit
[2:09:11] more broadly about Axon and the
[2:09:12] system that we have and where
[2:09:14] there are other systems and
[2:09:16] Aibas technology that we're
[2:09:18] using within that Axon platform
[2:09:19] if that's ok I'm obviously a
[2:09:20] bulk of the what we'll talk
[2:09:20] about is
[2:09:23] is around the fleet re andlPR
[2:09:25] but the first thing to talk
[2:09:27] about again is like the
[2:09:29] different products that we have
[2:09:31] with Axon. so the the one that I
[2:09:33] think that is most compelling is
[2:09:35] and we're about to turn on
[2:09:36] again actually I don't think any
[2:09:37] of these are actually
[2:09:38] functioning yet
[2:09:40] other than the actual incar
[2:09:42] camera system but translation
[2:09:45] there are some media coverage
[2:09:46] down and I believe it was down
[2:09:47] in Parker with aarker has an
[2:09:49] axon body worn camera and they
[2:09:51] were able to show the video of a
[2:09:53] of a police officer on a traffic
[2:09:54] stop having a conversation with
[2:09:55] a gentleman that was speaking in
[2:09:58] Mandarin to my knowledge I don't
[2:09:59] think I have a trooper and
[2:10:01] anywhere in the state that
[2:10:02] speaks Mandarin I might have one
[2:10:03] or two that I just don't know
[2:10:05] about. but our intention is
[2:10:06] to turn on this technology so
[2:10:07] that we can have live
[2:10:11] interpretation like right
[2:10:12] there on the roadside and so
[2:10:13] that's I think that's a game
[2:10:15] changer for us that allows us to
[2:10:16] have more thorough
[2:10:18] investigations have more
[2:10:19] complete conversations on the
[2:10:21] side of the road which is a
[2:10:23] that's going to be a really
[2:10:25] really big win for us. The
[2:10:27] other thing that we'll use
[2:10:28] within the axon system also
[2:10:29] mostly off the body worn camera
[2:10:33] is that transcription process
[2:10:34] and so we'll be able to this
[2:10:37] this kind of sometimes gets
[2:10:38] overlapped with what
[2:10:39] they call I think it's called
[2:10:43] draft one axon draft one that is
[2:10:45] their whole reporting system and
[2:10:47] the patrol is not currently
[2:10:48] using that and so there are some
[2:10:50] agencies around the country that
[2:10:51] will actually write an entire
[2:10:55] police report based on what
[2:10:57] the body worn cameras record and
[2:11:00] do that and I not a popular
[2:11:01] opinion amongst everybody but
[2:11:03] I've been very very hesitant
[2:11:05] about turning that on for us. I
[2:11:07] have concerns about taking away
[2:11:09] the human in the context of this
[2:11:10] hearing the human in the loop to
[2:11:10] actually
[2:11:13] describe what they saw like so
[2:11:14] many of the decisions that
[2:11:15] police officers are being asked
[2:11:17] to make in split seconds and all
[2:11:18] of the case law that supports
[2:11:20] those decisions are based on
[2:11:21] that officer's ability to
[2:11:23] articulate their action in that
[2:11:25] moment. a great example that
[2:11:27] I give is because obviously the
[2:11:28] colorado state patrol does a lot
[2:11:30] of work on impaired driving
[2:11:31] and you and I could watch the
[2:11:33] same video of the same person
[2:11:35] and because of my training and
[2:11:37] my experience they're a cues
[2:11:39] that I can pick up on that are
[2:11:40] an indicator that I've been
[2:11:40] trained or an indicator of
[2:11:41] impairment
[2:11:43] and I'm not confident that that
[2:11:45] the system is ready to do all of
[2:11:47] those things and on the most
[2:11:48] significant cases where there's
[2:11:49] in fact like an officer involved
[2:11:50] shooting or something like that.
[2:11:52] I don't want that to be to be
[2:11:54] the first time that a trooper is
[2:11:55] articulating what they did and
[2:11:56] why they did it. so we want to
[2:11:57] make sure that they're in the
[2:11:58] habit of articulating those
[2:12:00] actions that being said
[2:12:02] transcription for a lot of
[2:12:04] other reasons can be a very very
[2:12:05] valuable tool in a time saver
[2:12:08] for us and so we are starting to
[2:12:10] turn that on not widely
[2:12:10] spread used but again testing
[2:12:11] this
[2:12:13] to talk about interviews to
[2:12:15] just transcribe not a completed
[2:12:17] report and Axon has a
[2:12:19] transcription service built into
[2:12:20] its software that we can use to
[2:12:21] do that whether that's an
[2:12:22] interview whether it's an
[2:12:24] internal interview or out there
[2:12:25] on the street and it looks like
[2:12:26] there might be a question on
[2:12:27] that. I have a question what was
[2:12:31] the name of the setting that is
[2:12:32] basically cause I think it's a
[2:12:35] vendor setting if I'm
[2:12:36] understanding you correctacim
[2:12:38] will allow you to turn on or
[2:12:40] off? do you want to create a
[2:12:40] police report from this
[2:12:43] Currently we have it set to no
[2:12:46] what's the name of that field or
[2:12:48] that program did you say fleet
[2:12:50] or something like that you
[2:12:51] madamir it's don't know if it's
[2:12:53] actually a a flip of a switch
[2:12:55] cause way Axon works sometimes
[2:12:57] it's a flip switch sometimes
[2:12:58] it's not but the system is
[2:12:59] called draft oneraft yeah and
[2:13:02] it's as I understand it it's
[2:13:02] more connected to
[2:13:05] their like RMS system that
[2:13:06] they have that we also don't use
[2:13:11] but isrmS stand for? Records
[2:13:13] managementment system and we
[2:13:14] don't use the record there are
[2:13:15] records management system. Do we
[2:13:16] have our own internal
[2:13:19] proprietary RMS thank you
[2:13:21] madamir and with the help of the
[2:13:23] JTC last session and the
[2:13:24] joint budget committee we are
[2:13:26] anticipating going to RFP later
[2:13:29] this fiscal year for a new RMS
[2:13:30] for us because we don't really
[2:13:32] have one don't remember that but
[2:13:34] I'm going to trust you on that
[2:13:36] there were a lot of I know I
[2:13:38] know I know perfect. Well
[2:13:38] rett Pascal
[2:13:43] thank you madam chairir. just
[2:13:44] wondering is there a particular
[2:13:46] reason you guys don't just turn
[2:13:47] this one on and you want to use
[2:13:48] a different records management
[2:13:50] system or what's the thinking
[2:13:53] there or is this a potential RFP
[2:13:55] before you answer that I'm super
[2:13:57] glad we keep that internally
[2:13:59] because I personally don't love
[2:14:00] the fact thatacxam would have
[2:14:03] all of our records so I'm glad
[2:14:05] that we use an internal records
[2:14:07] management system but please
[2:14:09] Chief Packard answer. thank you
[2:14:10] madamir andrett Pasco
[2:14:14] we don't know what that's gonna
[2:14:15] look like yet so we're in the
[2:14:16] process of drafting our
[2:14:18] requirements and of axon is one
[2:14:20] of the organizations that
[2:14:23] provides a bid or a proposal
[2:14:25] for us to consider we'll do that
[2:14:26] but whether or not we go with
[2:14:28] them or not will be based on all
[2:14:29] of the different settings and
[2:14:31] there's a requirements I should
[2:14:32] say that will be vast
[2:14:36] Rep to tone
[2:14:41] thank you madam chairir. one
[2:14:44] question I had is about sharing
[2:14:44] information with
[2:14:48] with other organizations and
[2:14:51] that's been something that has
[2:14:53] been topic at the legislature
[2:14:57] is you know flock and and a
[2:14:59] on the all operate basically the
[2:15:00] same way
[2:15:04] of just sharing this data and
[2:15:05] even selling it to
[2:15:09] other places without a
[2:15:11] warrant and that's something
[2:15:13] that I'm concerned about.
[2:15:14] what, what are your policies on
[2:15:14] that?
[2:15:19] cheap packard thank you madamir
[2:15:22] andpresentative to tone so
[2:15:23] with Axon
[2:15:26] the sharing of data that we have
[2:15:27] on and talking about like what
[2:15:29] we've addressed so far and and
[2:15:31] we can move into the LPr
[2:15:33] conversation which might be more
[2:15:35] in step with the the root of the
[2:15:38] question rep to tone but
[2:15:39] generally speaking the data that
[2:15:41] we collect with Axon stays
[2:15:44] within the the ecosystem within
[2:15:45] axon and then we have total
[2:15:46] control of where that goes.
[2:15:50] so right now for example a body
[2:15:53] worn camera review or or
[2:15:55] video or or even a video of a
[2:15:57] fleet from fleet 3 there's an
[2:15:59] incar camera where that might go
[2:16:00] there's a couple of different
[2:16:02] options obviously we will
[2:16:03] share that with other partner
[2:16:05] agencies because of we're on the
[2:16:09] same case and so a great example
[2:16:10] I stopped with a car crash an
[2:16:11] injury crash in a municipal
[2:16:13] jurisdiction a few months ago.
[2:16:15] my body worn camera was on the
[2:16:16] patrol was not investigating
[2:16:17] that in
[2:16:19] ci d ent I just happened to be
[2:16:21] there. my body worn camera video
[2:16:23] then was shared and sent over to
[2:16:25] the investigating agency so they
[2:16:26] can ingest that into their
[2:16:27] records and presumably that
[2:16:29] ultimately went to a district
[2:16:30] attorney's office for some sort
[2:16:30] of prosecution
[2:16:33] we would do the same thing and
[2:16:35] that's probably the number one
[2:16:38] recipient of our all of our
[2:16:40] video which is the most of our
[2:16:43] ecosystem is videobased most of
[2:16:45] it our primary receiver of
[2:16:47] that goes to the district
[2:16:49] attorney's office and there is a
[2:16:50] sharing function in the system
[2:16:52] that we can provide that like
[2:16:53] directly to them so that they
[2:16:55] can have that for their for for
[2:16:57] their responsibilities the
[2:16:59] only other place that it might
[2:17:00] be shared is when it comes
[2:17:02] under you know things like co
[2:17:04] request or CCGRA analysis where
[2:17:06] members of the public are
[2:17:08] requesting video and we'll go
[2:17:09] through the factors outlined in
[2:17:11] law about what should and what
[2:17:12] can and can be cannot be
[2:17:13] released what should be redacted
[2:17:15] and what cannot be redacted
[2:17:17] video reaction as a whole
[2:17:17] another that's a huge
[2:17:21] laborious conversation as well
[2:17:24] but those things would would
[2:17:26] go through and of course as
[2:17:28] pursuant to law like we make
[2:17:29] sure that folks attach that
[2:17:31] they're not ingesting that for
[2:17:33] pecuniary gain like if they're
[2:17:34] doing that it's for education
[2:17:35] for some other reason that would
[2:17:38] be accessible via CCGRA or
[2:17:38] Cora request.
[2:17:47] OK, very good. Please continue.
[2:17:50] OK, moving on to the
[2:17:51] conversation specifically around
[2:17:52] fleet re
[2:17:55] so again I'll just tell you
[2:17:57] that we have not turned this on
[2:17:59] yet. so but I I wanna take a
[2:18:01] little opportunity to also to
[2:18:02] speak more holistically about
[2:18:05] LPRs and how we're doing that
[2:18:07] we do have ALPRs at the colorado
[2:18:08] sea patrol. we've actually had
[2:18:10] them functioning for well over a
[2:18:11] decade and right now we are
[2:18:14] eva evaluating whether or not to
[2:18:15] activate the ALpR function
[2:18:16] within fleet re.
[2:18:19] and so going through some of
[2:18:21] the the considerations here.
[2:18:24] data sharing is one specifically
[2:18:26] to fleet 3 data sharing anything
[2:18:28] any license plate read that
[2:18:30] happens in a in a cholera sea
[2:18:33] patrol run system stays within
[2:18:37] the patrols ecosystem it does
[2:18:38] not automatically go out
[2:18:39] specifically with an axon. We
[2:18:41] talk about some of the other
[2:18:42] ALpR systems that we have we
[2:18:45] have two other vendors one of
[2:18:46] them we're not sharing
[2:18:49] anything anywhere yet. we
[2:18:51] actually made a decision in 2025
[2:18:52] that we turned off all sharing
[2:18:55] agreements so we were the only
[2:18:56] people that had access to that
[2:18:56] information
[2:18:59] if another agency had an
[2:19:01] investigatory investigatory need
[2:19:03] to see if we had contact with
[2:19:04] there's something along those
[2:19:06] lines that an agent or or an
[2:19:09] agent like a a police officer
[2:19:10] or a detective or something
[2:19:12] could call us and talk to a
[2:19:14] state patrol employee and said
[2:19:15] hey we're looking for this
[2:19:17] information on this thing and
[2:19:19] then our process is to take log
[2:19:20] like this is the case number
[2:19:22] this is the crime that there's
[2:19:23] an internal vetting before we
[2:19:25] would ever look in providing
[2:19:26] the information related to that
[2:19:26] but an important
[2:19:29] distinction here is nobody
[2:19:31] has their own access to our
[2:19:33] systems it's a it's a phone
[2:19:34] conversation they have to have
[2:19:35] with one of our folks before
[2:19:37] they might have access to that
[2:19:41] the the value of fleet re
[2:19:43] with Axon is
[2:19:45] it's multilane scanning we can
[2:19:46] load hot lists into that
[2:19:50] and that's that's the huge
[2:19:53] value for fleet re is we have
[2:19:54] the tech we have the opportunity
[2:19:55] to have a hot list that comes
[2:19:58] from NCIC or CCIC in in layman's
[2:20:00] term that is the system where
[2:20:02] vehicles are are noted as being
[2:20:03] stolen or for some other reason
[2:20:05] that they're a vehicle of
[2:20:07] interest and we are our system
[2:20:09] is set up to well can be set up
[2:20:11] to refresh that hot list every4
[2:20:13] hours and so that would come in
[2:20:16] and so the that's there
[2:20:16] and so as troopers are driving
[2:20:19] down the road if they receive a
[2:20:21] hit on their ALPR they will
[2:20:23] they'll hit that and then they
[2:20:25] the hit will appear they can
[2:20:26] look for the vehicle that is
[2:20:27] supposed to evidently close to
[2:20:29] them because it hit on their
[2:20:31] incar camera. they'll find that
[2:20:33] vehicle and then they'll go
[2:20:34] through their own manual
[2:20:35] verification process number one
[2:20:37] that is the car that is the
[2:20:39] plate they'll obtain a
[2:20:40] clearance independently back
[2:20:41] through the system to make sure
[2:20:43] it didn't fall off NCIC or CCIC
[2:20:45] in the last four hours to make
[2:20:46] sure that it is still a wanted
[2:20:46] vehicle
[2:20:49] for whatever reason that might
[2:20:51] be and then in the initiate
[2:20:53] whatever tactic they might do to
[2:20:55] to make a traffic stop to call
[2:20:56] in additional units or whatever
[2:20:57] that might be based on the
[2:20:59] nature of the hit. there's
[2:21:01] other opportunities to load into
[2:21:03] the system like vehicles of
[2:21:07] interest so for example a an
[2:21:08] ammber alert comes out quickly
[2:21:09] we can have the opportunity to
[2:21:12] put in a hot list that goes
[2:21:13] in the system independently
[2:21:15] within the patrol our policy is
[2:21:16] that we come through an
[2:21:19] investigative investigative
[2:21:20] group to make sure this is
[2:21:21] something we are ok with going
[2:21:25] in live into our system. if
[2:21:26] it's not touched otherwise that
[2:21:27] is a thirtyday life cycle so
[2:21:29] it's in there for 30 days and
[2:21:32] then drops back off. there's
[2:21:33] a review process that goes
[2:21:37] through that. The the audit
[2:21:38] process so we currently have an
[2:21:40] ALPr coordinator that works for
[2:21:40] us it's a position
[2:21:43] and that person ensures that all
[2:21:45] of the folks are trained
[2:21:47] appropriately to use that that
[2:21:49] also goes through and audits the
[2:21:50] logs if there's a query of
[2:21:51] license plate reads to make sure
[2:21:52] that there's appropriate
[2:21:53] justification provided for each.
[2:21:55] if there's not that's when
[2:21:56] investigation can start to go
[2:21:57] through and to validate whatever
[2:21:59] look that might whatever reason
[2:22:01] for that query might have been
[2:22:02] if it was a clerical error or if
[2:22:03] it was in fact a violation of
[2:22:04] policy or something along those
[2:22:06] lines so that person can do that
[2:22:09] I'm gonna go through some of
[2:22:11] your questions here on ALPRs as
[2:22:12] well
[2:22:15] but you talked about
[2:22:17] flagged do you how the question
[2:22:18] was how is flagged vehicle data
[2:22:18] retained
[2:22:21] and so again that is through the
[2:22:24] NCIC or CCI system that fourho
[2:22:27] update there's also a4
[2:22:29] there's there's a period of time
[2:22:30] in NCIC for a record to stay
[2:22:33] valid. I think it's4 years
[2:22:35] but once the vehicle is found
[2:22:37] that it will drop off of what's
[2:22:39] called a locate is done in that
[2:22:41] NCIc or CCIC system once the
[2:22:44] locate is done then the next
[2:22:47] refresh of the ALPr system will
[2:22:48] drop that off of the hit list. I
[2:22:48] already
[2:22:51] mentioned to you the 30 day rule
[2:22:53] for internal general or I'm
[2:22:55] sorry internally generated
[2:22:57] vehicles of interest I
[2:22:59] suspect that question is also
[2:23:00] more about the actual license
[2:23:03] plate reads for fleet re we
[2:23:04] haven't made that decision yet
[2:23:06] and and we are in fact I had a
[2:23:07] conversation this morning about
[2:23:09] where we are in this process and
[2:23:11] while we've been using ALPRs
[2:23:13] in a more limited version
[2:23:15] specifically an investigative
[2:23:17] areas like going through with
[2:23:18] more cameras we want to make
[2:23:18] sure that everybody
[2:23:21] has sufficient training the
[2:23:22] policy is where it needs to be
[2:23:23] and we've made thoughtful
[2:23:25] decisions about things like
[2:23:27] sharing retention and all of
[2:23:29] those things right now the
[2:23:30] conversation we're thinking
[2:23:32] about is for fleet 3 that system
[2:23:34] will probably stay within Axon
[2:23:36] and it's it's a different value
[2:23:37] than for example a fixed
[2:23:39] location and then it would be
[2:23:41] for the mobileca mobile reads
[2:23:43] but a lot of these decisions are
[2:23:44] yet to be made. Our current
[2:23:46] policy for our current systems
[2:23:48] is a license plate read stays in
[2:23:48] our system
[2:23:51] for 180 days and there's a lot
[2:23:53] of conversation about that and
[2:23:54] I anticipate having that
[2:23:55] conversation lots more over the
[2:23:57] next several weeks and months
[2:24:00] but it's a our we are currently
[2:24:03] set at 180 days and then that is
[2:24:05] it the system drops off and our
[2:24:06] auditor our our internal auditor
[2:24:07] also goes there and verifies
[2:24:08] that
[2:24:11] to be clear on the data sharing
[2:24:15] component to this we have we
[2:24:16] should internally with other
[2:24:16] Department of Public Safety
[2:24:17] entities they have access to
[2:24:20] that we're also moving down
[2:24:21] the path that we have redone an
[2:24:23] MOU to make sure that it's with
[2:24:25] current law and retention and
[2:24:27] reasoning for searching
[2:24:27] specifically around the Colorado
[2:24:29] law about information about
[2:24:31] immigration enforcement to make
[2:24:32] sure it won't be used for that.
[2:24:34] each agency is required to
[2:24:37] reup and sign their MOU with us
[2:24:38] and that will be limited
[2:24:38] initially
[2:24:41] to agencies that we partner with
[2:24:42] through our auto theft task
[2:24:43] forcece so those are capppaw
[2:24:45] funded entities the Colorado
[2:24:46] Auto Theft Prevention Authority
[2:24:49] and then what we call battle
[2:24:51] is is a is an auto theft task
[2:24:53] force so the battle task force
[2:24:55] works so those agencies that are
[2:24:57] involved in that which have
[2:24:59] their own level of because of
[2:25:00] their engagement with Capaw and
[2:25:01] battle their own level of like
[2:25:03] contractual agreements and and
[2:25:05] acknowledgement of law and then
[2:25:07] an added layer with the MOU
[2:25:08] specifically within the A
[2:25:13] L p R data. yes ma'am. OK so
[2:25:14] you're saying that
[2:25:21] every agency must sign an MOU
[2:25:25] with CSP through the cat paw is
[2:25:28] that every agency that is
[2:25:33] using ALPr cameras or accent
[2:25:37] cameras or just people that are
[2:25:38] a part of the cat paw. I'm
[2:25:40] trying to understand what you
[2:25:42] mean like what who do you have
[2:25:43] MU's with?
[2:25:45] thank you madam chair and so the
[2:25:47] answer is right now I'm not sure
[2:25:48] we have one all the way
[2:25:51] completed yet but the initial
[2:25:53] offer of that MOU is with other
[2:25:54] agencies that are affiliated
[2:25:55] with battle the the auto theft
[2:25:58] task force with the cat paw. so
[2:25:59] not beyond that at that point
[2:26:01] there was a specific question
[2:26:02] about federal government
[2:26:03] agencies and we're not currently
[2:26:05] sharing with any federal entity
[2:26:06] that data as well.
[2:26:12] and then do you know the so
[2:26:15] yeah 180 days you're still well
[2:26:17] within the range we have nothing
[2:26:19] on the books that's any smaller
[2:26:22] than 3 years requirement. so
[2:26:25] a half a year is fine. I do
[2:26:26] think we'll be having having
[2:26:28] some conversations about the
[2:26:31] length of that retention
[2:26:37] but this this this all
[2:26:39] this all makes sense to me. I
[2:26:40] don't really have questions. I
[2:26:41] do think that rep to tone
[2:26:45] I think yes rep vice chair to
[2:26:46] tone you're up.
[2:26:47] thank you ma'amirir.
[2:26:53] so I I would be curious to to
[2:26:55] know what your policies are
[2:26:56] about abusing
[2:27:00] data cause I I know I've seen a
[2:27:00] lot of
[2:27:04] cases where officers have been
[2:27:05] using
[2:27:08] these ALPRs to track people
[2:27:13] ex's and people of interest
[2:27:14] that they just want to know
[2:27:15] where they are
[2:27:18] and I would like to know that
[2:27:19] you know what's your policies
[2:27:21] are and I would hope that you
[2:27:22] would
[2:27:22] have quite a a
[2:27:26] a robust policy against that
[2:27:29] because I don't want to see
[2:27:31] that kind of thing abused
[2:27:32] because that you know
[2:27:34] these kinds of cameras can
[2:27:38] help and assist in a crime. I
[2:27:38] don't
[2:27:42] deny that that is a you know
[2:27:43] something that it's possible but
[2:27:46] abusing it is very easy to do
[2:27:47] and I want to make sure that
[2:27:48] you're preventing that
[2:27:53] there was a another thing I was
[2:27:54] gonna ask about
[2:27:57] when it comes to
[2:28:02] a license plate that's not read
[2:28:04] correctly and I've seen some of
[2:28:05] these cases where
[2:28:09] there's a family being pulled
[2:28:10] out of a vehicle
[2:28:10] and
[2:28:14] guns are being drawn by multiple
[2:28:15] officers with
[2:28:19] that putting these innocent
[2:28:20] people in a very difficult
[2:28:20] situation.
[2:28:23] What what kinds of policies do
[2:28:25] you have for these and
[2:28:29] does the system give you a full
[2:28:31] confidence rating in in the
[2:28:32] reading of the plate so you know
[2:28:33] that like
[2:28:33] it's
[2:28:38] you know it's it's very sure
[2:28:40] it's got the match or it's not
[2:28:40] quite as
[2:28:43] as good of a match and does that
[2:28:46] go into the response that you
[2:28:47] take
[2:28:49] on those particular stops
[2:28:51] because you know you don't wanna
[2:28:53] accidentally shoot an innocent
[2:28:55] person whose license plates one
[2:28:56] letter off because that would be
[2:28:56] a
[2:28:59] pretty embarrassing thing to
[2:29:00] have happen.
[2:29:03] yeah Chief Packard so two things
[2:29:05] one would you be willing or able
[2:29:07] to send us a copy of your
[2:29:10] policies for the organization
[2:29:13] within5 days. Is that OK?ritten
[2:29:14] your written policies
[2:29:17] thank you madamir yeah we can
[2:29:19] show you that policy and I think
[2:29:22] if I can't continue with well
[2:29:24] was going to say I also am
[2:29:26] curious about how often you're
[2:29:28] checking the error rate of your
[2:29:29] vendor contracts so yeah
[2:29:33] play that in with what reptonene
[2:29:34] just asked please.
[2:29:36] thank you madamir madam vice
[2:29:38] chairir so our first stop with
[2:29:40] I'll start with the conversation
[2:29:42] about what policy says about
[2:29:45] misuse of the system the
[2:29:47] language is in fact I think it's
[2:29:48] in the front first page of the
[2:29:50] policy does any misuse of this
[2:29:52] system is subject to you know
[2:29:53] personnel action under the state
[2:29:55] personnel board and obviously
[2:29:57] because of the rules within the
[2:29:59] personnel board that language is
[2:30:00] intentionally broad
[2:30:03] but I just offer hopefully with
[2:30:04] some degree of comfort
[2:30:07] for you in the process that
[2:30:09] we're currently undertaking
[2:30:10] with before we're turning on and
[2:30:13] implementing fleet re and
[2:30:15] this is a robust process because
[2:30:17] not only would it be
[2:30:17] embarrassing it would be
[2:30:19] unlawful and it would be
[2:30:20] horrible if someone lost the
[2:30:21] life because of an improper
[2:30:23] action that a member of our
[2:30:24] organization or any organization
[2:30:27] took based on ALPR reader or
[2:30:29] anything else and so we're going
[2:30:31] to go through a robust training
[2:30:33] program for each member that
[2:30:34] would have access to this and in
[2:30:34] fact our current
[2:30:35] policy states that if you have
[2:30:37] not been trained that you were
[2:30:39] not permitted to use it and
[2:30:40] that training talks about the
[2:30:43] process to go through to vet the
[2:30:44] reid as it shows up on the
[2:30:47] system and so there's a couple
[2:30:49] of different things that could
[2:30:51] happen here. there could be
[2:30:53] an improper entry into NCIc or
[2:30:55] CCIC and so that plate might
[2:30:56] actually be an accurate read
[2:30:59] but then we train our members
[2:31:01] to make decisions based on what
[2:31:03] they're seeing as well and make
[2:31:06] oper enforcement decisions and
[2:31:07] tactical decisions based on the
[2:31:09] totality of the circumstances so
[2:31:11] the license plate read whether
[2:31:13] it's correct or not is but one
[2:31:15] factor in this overall analysis
[2:31:18] of how we might respond to a s a
[2:31:20] specific incident and we'll
[2:31:21] train through as many of those
[2:31:22] scenarios as we possibly can
[2:31:25] with our members there is
[2:31:26] another scenario that might
[2:31:26] happen is
[2:31:29] like
[2:31:32] the maybe the best way to do
[2:31:35] this I I personally a personal
[2:31:37] story this was a long time
[2:31:38] ago when I was still a real
[2:31:40] trooper and I stopped on the
[2:31:41] side of the road with the
[2:31:43] vehicle and gave the plate
[2:31:45] license plate to the dispatcher
[2:31:47] who sent off all the alert
[2:31:49] tones on our radio this says hey
[2:31:50] this is a possible stolen
[2:31:51] vehicle. you know you need to
[2:31:52] you know
[2:31:55] handle cord right? well it
[2:31:57] was a mom and daughter with a
[2:31:59] flat tire and we are in the
[2:32:01] process of helping them change
[2:32:02] their flat tire when I received
[2:32:04] that information. Now obviously
[2:32:05] evaluating the entire scenario
[2:32:07] I'm like I wasn't too worried
[2:32:08] about the facts given at that
[2:32:09] moment. I was a different level
[2:32:11] of awareness and being more
[2:32:13] aware of my surroundings but
[2:32:14] talk through it well it turns
[2:32:17] out in that particular case
[2:32:18] it was a stolen license plate
[2:32:21] and not a vehicle and so and as
[2:32:22] I talked through the the the
[2:32:22] woman
[2:32:25] that was owned the vehicle that
[2:32:26] turns out their plate had been
[2:32:27] stolen and that's why that entry
[2:32:28] had been into the system
[2:32:30] and so the way we can work
[2:32:32] through those in an ALPR
[2:32:35] setting for example is an ALpr
[2:32:37] hit shows up on the computer we
[2:32:39] look at that and like I
[2:32:41] mentioned before locate the
[2:32:43] vehicle and then start
[2:32:44] looking what's around, what's
[2:32:45] going on are there other
[2:32:47] indicators of of potential
[2:32:50] criminal behavior rerun the
[2:32:51] license plate and so we can give
[2:32:53] that to our dispatcher you can
[2:32:55] the systems that our troopers
[2:32:56] have access to in their cars to
[2:32:57] say is this really a current
[2:33:01] stolen vehicle or whatever
[2:33:02] else and and then they can come
[2:33:03] back and say sometimes it could
[2:33:05] be a wanted person. well does
[2:33:07] the physical description of the
[2:33:08] person match the physical
[2:33:10] description of the warrant is it
[2:33:11] a stolen license plate or is it
[2:33:14] a stolen vehicle all of these
[2:33:16] things that can be evaluated
[2:33:17] while we're just driving
[2:33:18] alongside before we make an
[2:33:18] enforcement decision.
[2:33:21] and then once all of those
[2:33:23] facts are at play then that
[2:33:24] operational decision gets put
[2:33:26] into play and so there are many
[2:33:29] different steps along the way to
[2:33:30] make sure that there's an
[2:33:32] appropriate level response
[2:33:33] based on the situation that
[2:33:34] we're dealing with in that
[2:33:36] moment and that is all based
[2:33:38] on training and a review process
[2:33:41] that happens anytime that
[2:33:42] you know if there's a level of
[2:33:44] force that's used or the normal
[2:33:46] supervisor review of an arrest
[2:33:47] and investigation in some sort
[2:33:48] of incident so I spoke a lot
[2:33:51] there and madam vice chair maybe
[2:33:52] I'll pause there to make sure I
[2:33:52] answered all the questions you
[2:33:53] were looking for there.
[2:33:59] that's fine for now. thank you.
[2:34:00] very good
[2:34:05] Oh dewey, I'm sorry repelty
[2:34:11] thank you ma'amreau thank you
[2:34:13] and thank you sir for being here
[2:34:14] today. I do have a question
[2:34:16] it's very similar to rupted
[2:34:19] tonennes in in that that
[2:34:21] direction I'm gonna just tell
[2:34:23] you personally what happened and
[2:34:26] see if there's if you can
[2:34:28] help with the situation. I know
[2:34:30] it's a third party that does the
[2:34:32] the cameras for you know
[2:34:32] speeding and stuff like that
[2:34:35] that's around the area but
[2:34:38] myself I was I had to hurry
[2:34:38] around someone that was driving
[2:34:39] extreme
[2:34:41] ly dangerously and I just wanted
[2:34:42] to get away from them and in
[2:34:43] doing so I ended up getting a
[2:34:46] ticket because I went two miles
[2:34:49] or I was 11 miles over the speed
[2:34:50] you know to get around him and I
[2:34:51] got to the other camera because
[2:34:53] it's surveilling me from camera
[2:34:56] to camera and so it got to
[2:34:58] the other camera too soon and it
[2:35:00] just issued a ticket and
[2:35:01] normally in a situation like
[2:35:03] that an officer would pull you
[2:35:04] over you can explain to me hey
[2:35:05] that guy over there and that guy
[2:35:07] would be the one that the
[2:35:08] officer could go after you know
[2:35:09] the the the bad driver
[2:35:11] rather than the person trying to
[2:35:13] flee or get away from them.
[2:35:15] what type of policies do you
[2:35:16] guys have for something like
[2:35:17] that because trying to deal with
[2:35:19] this third party company that
[2:35:21] you that's being utilized for
[2:35:23] this is impossible you there is
[2:35:25] no way to object. there is no
[2:35:27] way to say hey you know there
[2:35:28] was a dangerous situation I had
[2:35:30] to get out of it I had to speed
[2:35:31] for a minute just to get out of
[2:35:33] it and it made me do 11 miles an
[2:35:36] hour over you know for or got me
[2:35:38] to the other camera faster than
[2:35:39] I should have, yes, but it
[2:35:41] was because of a certain
[2:35:43] situation. there's no way for
[2:35:45] our people for the for the
[2:35:47] drivers to be able to to say
[2:35:49] you know this is the situation
[2:35:50] the only way you can do it is go
[2:35:51] to court and then you have to
[2:35:53] pay the fees and then it ends up
[2:35:55] being rather than75 dollars fee
[2:35:56] which I had to pay. I just paid
[2:35:57] it just to get rid of it because
[2:35:58] that was during session I didn't
[2:35:59] have time for all that but other
[2:36:01] people have busy lives as well
[2:36:03] but they're just going to pay it
[2:36:04] rather than have to go to court,
[2:36:06] take time off of work, you know
[2:36:08] and hundreds of dollars later
[2:36:08] what kind of policy do you guys
[2:36:09] have
[2:36:11] or can you come up with a policy
[2:36:13] for people like in my situation
[2:36:15] where there was a very good
[2:36:17] reason as far as safety was
[2:36:19] concerned that I had to do make
[2:36:21] that decision what do you
[2:36:23] have for that for for then I can
[2:36:24] go back to my people and say hey
[2:36:27] you can you can do this cause
[2:36:28] right now it's a problem it's a
[2:36:28] big problem
[2:36:32] great really great example
[2:36:34] cheapef Packard thank you
[2:36:36] madamir and madam Representative
[2:36:39] I so I don't want to feel
[2:36:40] like I'm dodging your question,
[2:36:41] but the patrol does not use
[2:36:42] automated vehicle enforcement
[2:36:43] and so
[2:36:45] you can't get an automated
[2:36:47] citation from the Colorado State
[2:36:49] Patrol. Now there are other and
[2:36:51] there's obviously Cdot uses that
[2:36:53] technology in their
[2:36:54] enforcement in the managed
[2:36:55] lanes and some other scenarios
[2:36:57] and I so I don't have a
[2:36:58] strong awareness of the specific
[2:37:01] process that you have to go
[2:37:03] through to contest a violation
[2:37:05] that was received. I know a
[2:37:07] member of my family who I loved
[2:37:08] dearly and have've been married
[2:37:10] to for many years received
[2:37:14] one of those citations and
[2:37:15] of course I knew where she was
[2:37:17] and understood things and so we
[2:37:20] we paid the violation but my
[2:37:21] understanding is that there is
[2:37:22] instructions provided to that
[2:37:24] but
[2:37:27] honestly ma'am, I don't know
[2:37:28] that there's an opportunity to
[2:37:31] advocate for a justifiable
[2:37:32] reason to violate the law of the
[2:37:34] system and that and in the
[2:37:35] context of automated vehicle
[2:37:36] enforcement and again I'm not
[2:37:38] the expert here it's more about
[2:37:39] whether or not it was your
[2:37:40] vehicle and it was an actual
[2:37:41] violation of law as opposed to a
[2:37:43] justifiable exemption that you
[2:37:45] might be able to provide to a
[2:37:46] human being on the side of the
[2:37:46] road
[2:37:52] and it could have been Rep Kelty
[2:37:53] it could have been either an
[2:37:55] avis or an ALPR system
[2:37:58] you know
[2:37:59] and they're both
[2:38:02] yeah well I'm sorry what were
[2:38:03] you gonna say I didn't mean to
[2:38:03] interrupt
[2:38:06] apologize and I guess that
[2:38:08] the problem is is no one knows
[2:38:10] it's just a a thing mailed to
[2:38:13] you someone in the mail and you
[2:38:14] know everyone getting these
[2:38:16] tickets left and right and and
[2:38:18] there there is no justification
[2:38:20] to say a how are things
[2:38:21] calibratedb the certain
[2:38:23] situation was arried you know or
[2:38:25] see there was a fire there was a
[2:38:26] dangerous driver there was you
[2:38:28] know and there's a problem with
[2:38:31] these systems and I I guess it's
[2:38:32] apologize I assumed that was
[2:38:32] your guys's department that that
[2:38:33] that
[2:38:36] handled all that but again it's
[2:38:38] that nobody knows it is a nobody
[2:38:39] knows and each agency in
[2:38:42] Colorado who has these cameras
[2:38:44] manages their own contracts
[2:38:47] directly with the vendors and so
[2:38:50] repelty you've really touched
[2:38:53] the tip of the iceberg on this
[2:38:54] one for sure.
[2:38:57] so I appreciate that. Any other
[2:39:01] questions on the ALPR or the we
[2:39:02] were just talking kind of avis
[2:39:02] stuff too
[2:39:07] OK, seeing none if my folks
[2:39:10] online would hang online we
[2:39:11] are going
[2:39:14] to first of all I want to
[2:39:16] make sure we've exhausted all of
[2:39:17] the questions related to this
[2:39:19] because we will not be asking
[2:39:21] questions when we go into
[2:39:23] executive session related to
[2:39:25] anything but what is going to be
[2:39:27] presented to us so I just want
[2:39:29] to ask the committee to make
[2:39:30] sure we don't have any
[2:39:30] additional questions
[2:39:35] seeing none what we're going
[2:39:39] to do is go into executive
[2:39:40] session
[2:39:42] we
[2:39:46] have the ability to go into
[2:39:48] executive session for these last
[2:39:50] two issues because they are a
[2:39:52] matter related to capital
[2:39:56] complex security measures so
[2:40:00] this does require a 2/3 members
[2:40:02] approval so it will require4 of
[2:40:04] us to vote yes to go into
[2:40:07] executive session we will not
[2:40:09] have to vote to come out of
[2:40:10] executive session we will not be
[2:40:10] taking
[2:40:14] any formal action or polling we
[2:40:16] will just be listening to
[2:40:19] information and asking questions
[2:40:20] when we're done with the
[2:40:23] executive session, I will
[2:40:24] announce that we're done and
[2:40:26] then we will come back here and
[2:40:26] I will adjourn the meeting.
[2:40:30] does anyone have any questions
[2:40:31] before I entertain emotion?
[2:40:36] OK seeing none, I will entertain
[2:40:38] a motion reppasal
[2:40:42] I moved that the jointchnology
[2:40:44] committee meet in executive
[2:40:45] session in accordance with
[2:40:51] section 2464023A4 for the
[2:40:53] purpose of matters related to
[2:40:55] capital complex security
[2:40:56] measures
[2:40:58] and that's a proper motion Mr
[2:40:59] Gravy will you please pull the
[2:41:00] committee
[2:41:05] senators and representatives
[2:41:06] eisley is excusedba
[2:41:07] i
[2:41:09] kelty
[2:41:14] Pascal yes
[2:41:16] vice chair at home
[2:41:18] madam chair
[2:41:24] that passes at this time I am
[2:41:26] going to ask that the only
[2:41:28] people who will remain on the
[2:41:31] zoom or in the room are
[2:41:33] committee members, committee
[2:41:35] staff and department personnel
[2:41:46] does OIT support your work?
[2:41:48] OIT can stay
[2:41:57] OK
[2:55:06] OIT coming forward and we'll
[2:55:08] look forward to seeing you guys
[2:55:09] shortly. thank you so much.
[2:55:12] Thank you