(19)
(11) EP 4 433 941 A1

(12)

(43) Date of publication:
25.09.2024 Bulletin 2024/39

(21) Application number: 22789747.7

(22) Date of filing: 15.09.2022
(51) International Patent Classification (IPC): 
G06F 40/30(2020.01)
G06N 3/08(2023.01)
(52) Cooperative Patent Classification (CPC):
G06F 40/30; G06N 3/0455; G06N 3/0475; G06N 3/084; G06N 3/09; G06N 3/096; G06N 3/088
(86) International application number:
PCT/US2022/043564
(87) International publication number:
WO 2023/091226 (25.05.2023 Gazette 2023/21)
(84) Designated Contracting States:
AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR
Designated Extension States:
BA ME
Designated Validation States:
KH MA MD TN

(30) Priority: 16.11.2021 US 202163264163 P
18.05.2022 US 202217664031

(71) Applicant: Microsoft Technology Licensing, LLC
Redmond, WA 98052-6399 (US)

(72) Inventors:
  • HE, Pengcheng
    Redmond, Washington 98052-6399 (US)
  • GAO, Jianfeng
    Redmond, Washington 98052-6399 (US)
  • CHEN, Weizhu
    Redmond, Washington 98052-6399 (US)

(74) Representative: CMS Cameron McKenna Nabarro Olswang LLP 
Cannon Place 78 Cannon Street
London EC4N 6AF
London EC4N 6AF (GB)

   


(54) LANGUAGE-MODEL PRETRAINING WITH GRADIENT-DISENTANGLED EMBEDDING SHARING