IP Library › Granted Patent US 10,803,399
Granted Patent B1
US 10,803,399 · App. 14/850,195 · Granted Oct 13, 2020

Topic model based clustering of text data with machine learning utilizing interface feedback

Inventors: Raphael Cohen (Beer-Sheva, IL); Alon J. Grubshtein (Lehavim, IL); Ofri Masad (Beer-Sheva, IL)
Assignee: EMC IP Holding Company LLC
G06N20/00G06F16/93
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,803,399
App. No.
14/850,195
Filed
Sep 10, 2015
Granted
Oct 13, 2020
Kind
B1
Art Unit
2122
USPC
706/11
Abstract

An apparatus comprises a processing platform configured to implement a machine learning system for automated classification of documents comprising text data of at least one database. The machine learning system comprises a clustering module configured to assign each of the documents to one or more of a plurality of clusters corresponding to respective topics identified from the text data in accordance with at least one topic model, and an interface configured to present portions of documents assigned to a particular one of the clusters by the clustering module and to receive feedback regarding applicability of the corresponding topic to each of one or more of the presented portions on a per-portion basis. The topic model is updated based at least in part on the received feedback. The feedback may comprise, for example, selection of a confidence level for applicability of the topic to a given one of the presented portions.

Claims (57)

1. An apparatus comprising:

a processing platform configured to implement a machine learning system for automated classification of documents comprising text data of at least one database;

wherein the machine learning system comprises:

first program code that when executed by the processing platform implements a clustering module configured to assign each of the documents to one or more of a plurality of clusters corresponding to respective topics identified from the text data in accordance with at least one topic model; and

second program code that when executed by the processing platform selects example portions of given documents from the plurality of clusters and further implements an interface configured to present the example portions of the given documents, and to receive feedback regarding applicability of the corresponding topic to each of one or more of the example portions on a per-portion basis;

wherein the machine learning system is configured to generate said at least one topic model by automatically elevating the topics from the text data as respective sets of related terms;

wherein said at least one topic model is updated based at least in part on the received feedback;

wherein the text data comprises unstructured text data;

wherein the unstructured text data comprises unstructured service event data of a service events database and wherein the unstructured service event data comprises unstructured service request summaries including one or more problem summaries and one or more corresponding solution summaries;

wherein at least a portion of the first program code that when executed by the processing platform implements the clustering module comprises program code that when executed by the processing platform implements a topic model generator configured to generate the at least one topic model and wherein the feedback received via the interface is supplied to the topic model generator to update the at least one topic model such that additional documents are each automatically assigned to one or more of the plurality of clusters utilizing the updated at least one topic model; and

wherein the processing platform comprises one or more processing devices each comprising a processor coupled to a memory.

2. The apparatus of claim 1 wherein the machine learning system further comprises an additional interface permitting entry of labels for respective ones of the clusters corresponding to respective topics.

3. The apparatus of claim 1 wherein the machine learning system comprises:

a back end portion; and

a front end portion coupled to the back end portion;

the back end portion comprising a big data analytics system implementing machine learning functionality of the machine learning system including at least part of the clustering module;

the front end portion implementing at least part of the interface.

4. The apparatus of claim 3 wherein the big data analytics system of the back end portion comprises a massively parallel processing (MPP) database having an associated library of scalable in-database analytics functions and wherein the front end portion comprises a web server implemented utilizing at least one virtual machine.

5. The apparatus of claim 1 wherein said at least one database is updated to indicate the assignment of a given one of the documents to one or more of the clusters.

6. The apparatus of claim 1 wherein the topic model generator provides lists of topics at least a subset of which are elevated as respective sets of related terms from the text data.

7. The apparatus of claim 6 wherein the topic model generator is configured to implement latent Dirichlet allocation (LDA).

8. The apparatus of claim 1 wherein at least a subset of the example portions presented via the interface comprise respective sentences.

9. The apparatus of claim 1 wherein the interface is configured to permit selection of a particular confidence level for the applicability of the topic to a given one of the example portions.

10. The apparatus of claim 1 wherein the interface is configured to permit selection of an alternative topic for a given one of the example portions and wherein the assignment of the associated document to one or more clusters is adjusted by the clustering module in accordance with the selection of the alternative topic, the alternative topic being selected from a listing of a plurality of selectable alternative topics provided for the presented portion by the interface.

11. An information processing system comprising the apparatus of claim 1 .

12. A method comprising:

obtaining documents comprising text data of at least one database;

assigning each of the documents to one or more of a plurality of clusters corresponding to respective topics identified from the text data in accordance with at least one topic model in which the topics are automatically elevated from the text data as respective sets of related terms;

selecting example portions of given documents from the plurality of clusters;

presenting via an interface the example portions of the given documents;

receiving via the interface feedback regarding applicability of the corresponding topic to each of one or more of the example portions on a per-portion basis; and

updating said at least one topic model based at least in part on the received feedback;

wherein the text data comprises unstructured text data;

wherein the unstructured text data comprises unstructured service event data of a service events database and wherein the unstructured service event data comprises unstructured service request summaries including one or more problem summaries and one or more corresponding solution summaries;

wherein the feedback received via the interface is utilized to update the at least one topic model such that additional documents are each automatically assigned to one or more of the plurality of clusters utilizing the updated at least one topic model; and

wherein said obtaining, assigning, selecting, presenting, receiving and updating are performed by a processing platform comprising one or more processing devices.

13. The method of claim 12 further comprising permitting selection via the interface of a particular confidence level for the applicability of the topic to a given one of the example portions.

14. The method of claim 12 further comprising:

permitting selection via the interface of an alternative topic for a given one of the example portions; and

adjusting the assignment of the associated document to one or more clusters in accordance with the selection of the alternative topic.

15. The method of claim 14 wherein permitting selection via the interface of an alternative topic comprises permitting selection of the alternative topic from a listing of a plurality of selectable alternative topics provided for the given one of the example portions by the interface.

16. A non-transitory processor-readable storage medium having program code of one or more software programs embodied therein, wherein the program code when executed by at least one processing device of a processing platform causes the processing device:

to obtain documents comprising text data of at least one database;

to assign each of the documents to one or more of a plurality of clusters corresponding to respective topics identified from the text data in accordance with at least one topic model in which the topics are automatically elevated from the text data as respective sets of related terms;

to select a set of example portions of given documents from the plurality of clusters;

to present via an interface the example portions of the given documents;

to receive via the interface feedback regarding applicability of the corresponding topic to each of one or more of the example portions on a per-portion basis; and

to update said at least one topic model based at least in part on the received feedback;

wherein the text data comprises unstructured text data;

wherein the unstructured text data comprises unstructured service event data of a service events database and wherein the unstructured service event data comprises unstructured service request summaries including one or more problem summaries and one or more corresponding solution summaries; and

wherein the feedback received via the interface is utilized to update the at least one topic model such that additional documents are each automatically assigned to one or more of the plurality of clusters utilizing the updated at least one topic model.

17. The processor-readable storage medium of claim 16 wherein the program code when executed by at least one processing device further causes said processing device:

to permit selection via the interface of an alternative topic for a given one of the example portions; and

to adjust the assignment of the associated document to one or more clusters in accordance with the selection of the alternative topic.

18. The processor-readable storage medium of claim 16 wherein the program code when executed by at least one processing device further causes said processing device to provide an additional interface permitting entry of labels for respective ones of the clusters corresponding to respective topics.

19. The processor-readable storage medium of claim 16 wherein at least a subset of the example portions presented via the interface comprise respective sentences.

20. The processor-readable storage medium of claim 16 wherein the interface is configured to permit selection of a particular confidence level for the applicability of the topic to a given one of the example portions.

Assignments (10)
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053546/0001) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC IP HOLDING COMPANY LLC
Reel/Frame 071642/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Recorded May 20, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061753/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Recorded Apr 26, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061324/0001 →
RELEASE OF SECURITY INTEREST Recorded Nov 3, 2021
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
To: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL, L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; WYSE TECHNOLOGY L.L.C.
Reel/Frame 058216/0001 →
SECURITY AGREEMENT Recorded Apr 22, 2020
From: CREDANT TECHNOLOGIES INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 053546/0001 →
SECURITY AGREEMENT Recorded Mar 21, 2019
From: CREDANT TECHNOLOGIES, INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 049452/0223 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 29, 2016
From: EMC CORPORATION
To: EMC IP HOLDING COMPANY LLC
Reel/Frame 040203/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Reel/Frame 040136/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
Reel/Frame 040134/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 16, 2015
From: COHEN, RAPHAEL; GRUBSHTEIN, ALON J.; MASAD, OFRI
To: EMC CORPORATION
Reel/Frame 036577/0288 →
Cited By (6)
US 12,333,244 US 12,400,072 US 12,493,797 US 12,505,688 US 12,536,476 US 12,554,762