In a recent study evaluating how chatbots make loan suggestions for mortgage applications, researchers at Pennsylvania’s Lehigh University found something stark: there was clear racial bias at play.
With 6,000 sample loan applications based on data from the 2022 Home Mortgage Disclosure Act, the chatbots recommended denials for more Black applicants than identical white counterparts. They also recommended Black applicants be given higher interest rates, and labeled Black and Hispanic borrowers as “riskier.�
kAm(9:E6 2AA=:42?ED H6C6 g]dT >@C6 =:<6=J E@ 36 2AAC@G65 E92? q=24< 2AA=:42?ED H:E9 E96 D2>6 7:?2?4:2= AC@7:=6] p?5 2AA=:42?ED H:E9 �=@H� 4C65:E D4@C6D @7 ec_[ D2H 2 H:56C >2C8:? � H9:E6 2AA=:42?ED H6C6 2AAC@G65 hdT @7 E96 E:>6[ H9:=6 q=24< 2AA=:42?ED H6C6 2AAC@G65 =6DD E92? g_T @7 E96 E:>6]k^Am
kAm%96 6IA6C:>6?E 2:>65 E@ D:>F=2E6 9@H 7:?2?4:2= :?DE:EFE:@?D 2C6 FD:?8 px 2=8@C:E9>D[ >249:?6 =62C?:?8 2?5 =2C86 =2?8F286 >@56=D E@ DA665 FA AC@46DD6D =:<6 =6?5:?8 2?5 F?56CHC:E:?8 @7 =@2?D 2?5 >@CE8286D] %96D6 �3=24< 3@I� DJDE6>D[ H96C6 E96 2=8@C:E9>’D :??6C H@C<:?8D 2C6?’E EC2?DA2C6?E E@ FD6CD[ 92G6 E96 A@E6?E:2= E@ =@H6C @A6C2E:?8 4@DED 7@C 7:?2?4:2= 7:C>D 2?5 2?J @E96C :?5FDECJ 6>A=@J:?8 E96>[ D2:5 s@?2=5 q@H6?[ 2? 2DD:DE2?E 7:?E649 AC@76DD@C 2E {69:89 2?5 @?6 @7 E96 2FE9@CD @7 E96 DEF5J]k^Am
kAmqFE E96C6’D 2=D@ =2C86 A@E6?E:2= 7@C 7=2H65 EC2:?:?8 52E2[ AC@8C2>>:?8 6CC@CD 2?5 9:DE@C:42==J 3:2D65 :?7@C>2E:@? E@ 27764E E96 @FE4@>6D[ D@>6E:>6D :? 56EC:>6?E2=[ =:76\492?8:?8 H2JD]k^Am
kAmâ€�%96C6’D 2 A@E6?E:2= 7@C E96D6 DJDE6>D E@ <?@H 2 =@E 23@FE E96 A6@A=6 E96J’C6 :?E6C24E:?8 H:E9[â€� q@H6? D2:5] “x7 E96C6’D 2 32<65\:? 3:2D[ E92E 4@F=5 AC@A282E6 24C@DD 2 3F?49 @7 5:776C6?E :?E6C24E:@?D 36EH66? 4FDE@>6CD 2?5 2 32?<]â€k^Am
k9cmw@H 5@6D px 5:D4C:>:?2E6 :? 7:?2?46nk^9cm
kAms64:D:@?\>2<:?8 px E@@=D 2?5 =2C86 =2?8F286 >@56=D[ =:<6 E96 @?6D :? E96 {69:89 &?:G6CD:EJ 6IA6C:>6?E[ 2C6 36:?8 FD65 24C@DD 2 G2C:6EJ @7 :?5FDEC:6D[ =:<6 962=E9 42C6[ 65F42E:@?[ 7:?2?46 2?5 6G6? :? E96 ;F5:4:2= DJDE6>]k^Am
kAm|@DE >249:?6 =62C?:?8 2=8@C:E9>D 7@==@H H92E’D 42==65 4=2DD:7:42E:@? >@56=D[ >62?:?8 J@F 7@C>2==J 567:?6 2 AC@3=6> @C 2 BF6DE:@?[ 2?5 E96? J@F 7665 E96 2=8@C:E9> 2 D6E @7 :?AFED DF49 2D 2 =@2? 2AA=:42?E’D 286[ :?4@>6[ 65F42E:@? 2?5 4C65:E 9:DE@CJ[ |:4926= (6==>2?[ 2 4@>AFE6C D4:6?46 AC@76DD@C 2E E96 &?:G6CD:EJ @7 |:49:82?[ 6IA=2:?65]k^Am
kAm%96 2=8@C:E9> DA:ED @FE 2 C6DF=E � 2AAC@G65 @C ?@E 2AAC@G65] |@C6 4@>A=6I 2=8@C:E9>D 42? 2DD6DD E96D6 724E@CD 2?5 56=:G6C >@C6 ?F2?465 2?DH6CD[ =:<6 2 =@2? 2AAC@G2= H:E9 2 C64@>>6?565 :?E6C6DE C2E6]k^Am
kAm|249:?6 =62C?:?8 25G2?46D :? C646?E J62CD 92G6 2==@H65 7@C H92E’D 42==65 566A =62C?:?8[ @C 4@?DECF4E:@? @7 3:8 ?6FC2= ?6EH@C<D E92E 42? =62C? 7C@> =2C86 2>@F?ED @7 52E2] qFE :7 px’D 3F:=56CD 5@?’E <66A @3;64E:G:EJ :? >:?5[ @C C6=J @? 52E2 D6ED E92E C67=64E 566A\C@@E65 2?5 DJDE6>:4 C24:D>[ C6DF=ED H:== C67=64E E92E]k^Am
kAm“x7 :E EFC?D @FE E92E J@F 2C6 DJDE6>2E:42==J >@C6 @7E6? >2<:?8 564:D:@?D E@ 56?J 4C65:E E@ 46CE2:? 8C@FAD @7 A6@A=6 >@C6 E92? J@F >2<6 E9@D6 HC@?8 564:D:@?D 23@FE @E96CD[ E92E H@F=5 36 2 E:>6 E92E E96C6’D 2 AC@3=6> H:E9 E96 2=8@C:E9>[â€� (6==>2? D2:5] “p?5 6DA64:2==J H96? E9@D6 8C@FAD 2C6 8C@FAD E92E 2C6 9:DE@C:42==J 5:D25G2?E2865]â€k^Am
kAmq@H6? H2D :?:E:2==J :?DA:C65 E@ AFCDF6 E96 {69:89 &?:G6CD:EJ DEF5J 27E6C 2 D>2==6C\D42=6 2DD:8?>6?E H:E9 9:D DEF56?ED C6G62=65 E96 C24:2= 5:D4C:>:?2E:@? 3J E96 492E3@ED]k^Am
kAm�(6 H2?E65 E@ F?56CDE2?5 :7 E96D6 >@56=D 2C6 3:2D65[ 2?5 :7 E96J’C6 3:2D65 :? D6EE:?8D H96C6 E96J’C6 ?@E DFAA@D65 E@ 36[� q@H6? D2:5[ D:?46 F?56CHC:E:?8 :D 2 C68F=2E65 :?5FDECJ E92E’D ?@E 2==@H65 E@ 4@?D:56C C246 :? 564:D:@?\>2<:?8]k^Am
kAmu@C E96 @77:4:2= DEF5J[ q@H6? 2?5 2 C6D62C49 E62> C2? E9@FD2?5D @7 =@2? 2AA=:42E:@? ?F>36CD @G6C D6G6C2= >@?E9D E9C@F89 5:776C6?E 4@>>6C4:2= =2C86 =2?8F286 >@56=D[ :?4=F5:?8 ~A6?px’D v!% b]d %FC3@ 2?5 v!% c[ p?E9C@A:4’D r=2F56 b $@??6E 2?5 ~AFD 2?5 |6E2’D {=2>2 b\gq 2?5 b\f_q]k^Am
kAmx? @?6 6IA6C:>6?E[ E96J :?4=F565 C246 :?7@C>2E:@? @? 2AA=:42E:@?D 2?5 D2H E96 5:D4C6A2?4:6D :? =@2? 2AAC@G2=D 2?5 >@CE8286 C2E6D] x? @E96C[ E96J :?DECF4E65 E96 492E3@ED E@ “FD6 ?@ 3:2D :? >2<:?8 E96D6 564:D:@?D]� %92E 6IA6C:>6?E D2H G:CEF2==J ?@ 5:D4C6A2?4:6D 36EH66? =@2? 2AA=:42?ED]k^Am
kAmqFE :7 C246 52E2 :D?’E 4@==64E65 :? >@56C? 52J =6?5:?8[ 2?5 2=8@C:E9>D FD65 3J 32?<D 2C6 :?DECF4E65 E@ ?@E 4@?D:56C C246[ 9@H 5@ A6@A=6 @7 4@=@C 6?5 FA 86EE:?8 56?:65 >@C6 @7E6?[ @C @776C65 H@CD6 :?E6C6DE C2E6Dn q642FD6 >F49 @7 @FC >@56C?\52J 52E2 :D :?7=F6?465 3J 5:DA2C2E6 :>A24E[ @C E96 :?7=F6?46 @7 DJDE6>:4 C24:D>[ q@H6? D2:5]k^Am
kAm%9@F89 2 4@>AFE6C H2D?’E 8:G6? E96 C246 @7 2? 2AA=:42?E[ 2 3@CC@H6C’D 4C65:E D4@C6[ H9:49 42? 36 :?7=F6?465 3J 5:D4C:>:?2E:@? :? E96 =23@C 2?5 9@FD:?8 >2C<6ED[ H:== 92G6 2? :>A24E @? E96:C 2AA=:42E:@?] $@ >:89E E96:C +x! 4@56[ @C E96 4C65:E D4@C6D @7 @E96C >6>36CD @7 E96:C 9@FD69@=5[ 2== @7 H9:49 4@F=5 92G6 366? :?7=F6?465 3J E96 9:DE@C:4 C24:DE AC24E:46 @7 C65=:?:?8[ @C C6DEC:4E:?8 =6?5:?8 E@ A6@A=6 :? A@@C 2?5 ?@?H9:E6 ?6:893@C9@@5D]k^Am
kAm|249:?6 =62C?:?8 2=8@C:E9>D 2C6?’E 2=H2JD 42=4F=2E:?8 E96:C 4@?4=FD:@?D :? E96 H2J E92E 9F>2?D >:89E :>28:?6[ q@H6? D2:5] %96 A2EE6C?D :E :D =62C?:?8 2AA=J E@ 2 G2C:6EJ @7 D46?2C:@D[ D@ :E >2J 6G6? 36 5:86DE:?8 C6A@CED 23@FE 5:D4C:>:?2E:@?[ 7@C 6I2>A=6 =62C?:?8 E92E q=24< A6@A=6 92G6 9:DE@C:42==J 925 H@CD6 4C65:E] %96C67@C6[ E96 4@>AFE6C >:89E D66 D:8?D E92E 2 3@CC@H6C :D q=24<[ 2?5 56?J E96:C =@2? @C @776C E96> 2 9:896C :?E6C6DE C2E6 E92? 2 H9:E6 4@F?E6CA2CE]k^Am
k9cm~E96C @AA@CEF?:E:6D 7@C 5:D4C:>:?2E:@?k^9cm
kAms64:D:@? >2<:?8 E649?@=@8:6D 92G6 364@>6 F3:BF:E@FD :? 9:C:?8 AC24E:46D @G6C E96 =2DE D6G6C2= J62CD[ 2D 2AA=:42E:@? A=2E7@C>D 2?5 :?E6C?2= DJDE6>D FD6 px E@ 7:=E6C E9C@F89 2AA=:42E:@?D[ 2?5 AC6\D4C66? 42?5:52E6D 7@C 9:C:?8 >2?286CD] {2DE J62C[ }6H *@C< r:EJ 3682? C6BF:C:?8 6>A=@J6CD E@ ?@E:7J 42?5:52E6D 23@FE E96:C FD6 @7 px 564:D:@?\>2<:?8 D@7EH2C6]k^Am
kAmqJ =2H[ E96 px E@@=D D9@F=5 36 AC@8C2>>65 E@ 92G6 ?@ @A:?:@? @? AC@E64E65 4=2DD6D =:<6 86?56C[ C246 @C 286[ 3FE D@>6 FD6CD 2==686 E92E E96J’G6 366? 5:D4C:>:?2E65 282:?DE 3J E96 2=8@C:E9>D 2?JH2J] x? a_a`[ E96 &]$] tBF2= t>A=@J>6?E ~AA@CEF?:EJ r@>>:DD:@? =2F?4965 2? :?:E:2E:G6 E@ 6I2>:?6 >@C6 4=@D6=J 9@H ?6H 2?5 6I:DE:?8 E649?@=@8:6D 492?86 E96 H2J 6>A=@J>6?E 564:D:@?D 2C6 >256] {2DE J62C[ E96 4@>>:DD:@? D6EE=65 :ED 7:CDE\6G6C px 5:D4C:>:?2E:@? 9:C:?8 =2HDF:E]k^Am
kAm%96 }6H *@C< 7656C2= 4@FCE 42D6 6?565 :? 2 Sbed[___ D6EE=6>6?E H96? EFE@C:?8 4@>A2?J :%FE@CvC@FA x?4] H2D 2==6865 E@ FD6 2? px\A@H6C65 9:C:?8 E@@= E92E C6;64E65 H@>6? 2AA=:42?ED @G6C dd 2?5 >6? @G6C e_] %H@ 9F?5C65 2AA=:42?ED C646:G65 E96 D6EE=6>6?E[ 2?5 :%FE@C 28C665 E@ 25@AE 2?E:\5:D4C:>:?2E:@? A@=:4:6D 2?5 4@?5F4E EC2:?:?8 E@ 6?DFC6 4@>A=:2?46 H:E9 6BF2= 6>A=@J>6?E @AA@CEF?:EJ =2HD[ q=@@>36C8 C6A@CE65 2E E96 E:>6]k^Am
kAmp?@E96C 2?E:\5:D4C:>:?2E:@? =2HDF:E :D A6?5:?8 :? r2=:7@C?:2 7656C2= 4@FCE 282:?DE px\A@H6C65 4@>A2?J (@C<52J] !=2:?E:77 s6C6< |@3=6J 2==686D 96 H2D A2DD65 @G6C 7@C >@C6 E92? `__ ;@3D E92E 4@?EC24E H:E9 E96 D@7EH2C6 4@>A2?J 3642FD6 96 :D q=24<[ @=56C E92? c_ 2?5 92D >6?E2= 962=E9 :DDF6D[ #6FE6CD C6A@CE65 E9:D DF>>6C] %96 DF:E 4=2:>D E92E (@C<52J FD6D 52E2 @? 2 4@>A2?J’D 6I:DE:?8 H@C<7@C46 E@ EC2:? :ED D@7EH2C6[ 2?5 E96 AC24E:46 5@6D?’E 244@F?E 7@C E96 5:D4C:>:?2E:@? E92E >2J C67=64E :? 7FEFC6 9:C:?8]k^Am
kAm&]$] ;F5:4:2= 2?5 4@FCE DJDE6>D 92G6 2=D@ 368F? :?4@CA@C2E:?8 564:D:@?\>2<:?8 2=8@C:E9>D :? 2 92?57F= @7 @A6C2E:@?D[ =:<6 C:D< 2DD6DD>6?E 2?2=JD:D @7 5676?52?ED[ 56E6C>:?2E:@?D 23@FE AC6EC:2= C6=62D6[ 5:G6CD:@?[ D6?E6?4:?8 2?5 AC@32E:@? @C A2C@=6]k^Am
kAm&E29 A2DD65 2 =2H 62C=:6C E9:D J62C E@ 4@>32E 6I24E=J E92E] wq bee[ DA@?D@C65 3J DE2E6 #6A] z2C:2??6 {:D@?366[ #\$JC24FD6[ 255C6DD6D E96 FD6 @7 2? 2=8@C:E9> @C 2 C:D< 2DD6DD>6?E E@@= D4@C6 :? 56E6C>:?2E:@?D 23@FE AC6EC:2= C6=62D6[ 5:G6CD:@?[ D6?E6?4:?8[ AC@32E:@? 2?5 A2C@=6[ D2J:?8 E92E E96D6 E649?@=@8:6D >2J ?@E 36 FD65 H:E9@FE 9F>2? :?E6CG6?E:@? 2?5 C6G:6H]k^Am
