Granted Patent B1

US 7,647,589 · App. 11/139,787 · Granted Jan 12, 2010

Abstract

Claims (155)

1. A method for running a virtual machine in a computing system, comprising:

(a) launching a virtual machine monitor (VMM) that uses a software debugger;

(b) launching a virtual machine (VM) that can natively run safe instructions;

(c) determining, at runtime, if the instruction is safe or potentially unsafe and (i) adjusting a context of the VM to enter an exception state during attempts to execute the potentially unsafe instructions, (ii) analyzing, at runtime and while running the control logic, a context of the VM and a set of previously executed instructions, and (iii) correcting a state of the VM if the state of the VM was damaged by a potentially unsafe instruction from a set of previously executed instructions;

(d) executing the safe instructions in a native mode of the VM; and

(e) activating control logic to process potentially unsafe instructions in the software debugger.

2. The method of claim 1 , further comprising:

(f) bypassing at least one potentially unsafe instruction.

3. The method of claim 2 , wherein step (f) further comprises:

generating bypassing code for the at least one potentially unsafe instruction; and

executing the bypassing code.

4. The method of claim 2 , wherein step (f) further comprises:

transferring control to the VM after processing the at least one potentially unsafe instruction in the VMM; and

restarting the at least one potentially unsafe instruction when that potentially unsafe instruction is safe in a current context of the VM.

5. The method of claim 2 , wherein step (f) further comprises:

using the VMM, adjusting a current context of the VM to ensure safe execution of the at least one potentially unsafe instruction in the VM;

transferring control from the VMM to the VM; and

executing the at least one potentially unsafe instruction in the adjusted context of the VM.

6. The method of claim 1 , wherein step (c) comprises analyzing the instruction prior to its execution.

7. The method of claim 1 , wherein step (c) comprises analyzing, at runtime and while running the control logic, a context of the VM to predict if a set of instructions to be executed in the future contains a potentially unsafe instruction.

8. A method for running a virtual machine in a computing system, comprising:

(a) launching a virtual machine monitor (VMM) that uses a software debugger;

(b) launching a virtual machine (VM) that can natively run safe instructions;

(c) executing the safe instructions in a native mode of the VM;

(d) activating control logic to process potentially unsafe instructions in the software debugger,

(e) analyzing a source page of guest code to identify the potentially unsafe instructions;

(f) determining, at runtime, if the instruction is safe or potentially unsafe;

(g) generating a target page of the guest code with breakpoints replacing the potentially unsafe instructions;

(h) executing instructions of the target page in a context of the VM;

(i) grouping the target Pages in a frame;

(i) executing intra-frame transfers of control in the context of the VM; and

(k) activating the software debugger to handle out-of-frame transfers of control.

9. The method of claim 8 , wherein any of an INT1, INT3 and INT(n) interrupts are used as the breakpoints.

10. The method of claim 8 , further comprising dynamically mapping each page of the frame used by guest OS as “present” in a page table entry (PTE).

11. The method of claim 10 , further comprising representing the frame as a plurality of page directory entries (PDE) that reflect an area of guest linear address space.

12. The method of claim 8 , wherein step (d) further comprises

analyzing a plurality of pages of guest code to identify the potentially unsafe instructions;

grouping the analyzed pages in a plurality of analyzed frames;

mapping several analyzed frames simultaneously into guest address space;

executing transfers of control between the mapped frames in the context of the VM; and

activating the software debugger to handle transfers of control between at least one mapped frame and at least one unmapped frame.

13. The method of claim 8 , wherein step (d) further comprises:

mapping the target page to a frame; and

establishing page protection on the corresponding source page to intercept attempts of the guest code to modify the corresponding source page.

14. The method of claim 13 , further comprising, upon detecting an attempt by the guest code to modify corresponding source page by means of a page fault:

unmapping the target page from the frame;

marking the target page as “changed” in a target page cache; and

removing protection from corresponding source page.

15. The method of claim 14 , further comprising, when mapping into the frame a target page from target pages cache, if a page is marked as “changed” and if analyzed address area was affected by those changes:

removing the target page from the target page cache;

initializing a new target page; and

starting analyzing routine.

16. The method of claim 15 , further comprising, upon detecting an attempts by the guest code to modify a corresponding source page by means of page fault:

unmapping the target page from the frame;

removing the target page from the target page cache; and

removing protection from corresponding source page.

17. The method of claim 8 , wherein step (d) further comprises:

mapping a first page of analyzed guest code to a frame; and

activating the software debugger by raising a page fault when an instruction of the first page attempts to transfer control to second page that is not mapped in the frame.

18. The method of claim 17 , further comprising mapping the second page in the frame such that subsequent attempts to transfer control to the second page will not raise the page fault.

19. The method of claim 1 , wherein step (e) further comprises transferring control to an exception handling routine of a guest operating system (OS) of the VM.

20. The method of claim 1 , wherein step (e) further comprises transferring control to an exception handling routine of a host operating system (OS).

21. The method of claim 1 , wherein step (e) further comprises transferring control to an exception handling routine of the VMM.

22. The method of claim 1 , wherein step (e) further comprises:

generating a target page filled with breakpoints;

replacing the breakpoints in the target page with the safe instructions of a source page of guest code, with address offsets preserved; and

executing instructions of the target page in a context of the VM.

23. The method of claim 1 , wherein step (e) further comprises:

generating a target page by replacing a potentially unsafe instruction of a source page of guest code with an instruction that calls a stub;

executing instructions of the target page in the context of the VM;

after calling the stub, determining whether the potentially unsafe instructions can be safely executed in the context of the VM; and

activating the software debugger to handle the potentially unsafe instructions incapable of being safely executed in the context of the VM.

24. The method of claim 1 , wherein step (e) further comprises resuming guest code execution if the unsafe instructions are safely executable in the context of the VM.

25. The method of claim 24 , wherein the generating step further comprises replacing potentially unsafe instructions with any of an INT1, INT3 and INT(n) interrupts that calls the stub.

26. A method for running a virtual machine in a computing system, comprising:

analyzing, at runtime, a source page of guest code to identify potentially unsafe instructions and (i) adjusting a context of the VM to enter an exception state during attempts to execute the potentially unsafe instructions, (ii) analyzing, at runtime and while running the control logic, a context of the VM and a set of previously executed instructions, and (iii) correcting a state of the VM if the state of the VM was damaged by a potentially unsafe instruction from a set of previously executed instructions;

generating a target page of the guest code with breakpoints replacing the potentially unsafe instructions;

transferring control from a virtual machine (VM) running the guest code to a software debugger of a virtual machine monitor (VMM) when an exception is raised at one of the breakpoints; and

bypassing at least one potentially unsafe instruction.

27. A computing system for running a virtual machine, comprising:

a virtual machine monitor (VMM) that uses a software debugger;

a virtual machine (VM) that executes safe instructions in a native mode; and

control logic that determines, at runtime, if the instruction is safe or potentially unsafe, and activates the software debugger in a VMM context to handle the potentially unsafe instructions, the control logic also (i) adjusting a context of the VM to enter an exception state during attempts to execute the potentially unsafe instructions, (ii) analyzing, at runtime and while running the control logic, a context of the VM and a set of previously executed instructions, and (iii) correcting a state of the VM if the state of the VM was damaged by a potentially unsafe instruction from a set of previously executed instructions

wherein the software debugger bypasses at least one of the potentially unsafe instructions.

28. The system of claim 27 , wherein the potentially unsafe instructions include instructions that cannot be safely executed in the context of the VM, and instructions that can cause unpredictable results in the context of the VM.

29. The system of claim 27 , wherein the software debugger restarts execution of the at least one potentially unsafe instruction when that potentially unsafe instruction is safe in the context of the VM.

30. The system of claim 27 , wherein the software debugger adjusts a current context of the VM to ensure safe execution of the at least one potentially unsafe instruction in the context of the VM.

31. The system of claim 27 , wherein the control logic analyzes a source page of guest code to identify the potentially unsafe instructions, and generates a target page of the guest code, with breakpoints replacing the potentially unsafe instructions in the target page, and wherein the VM executes instructions of the target page.

32. The system of claim 31 , wherein the breakpoints include any of an INT1, INT3, and INT(n) interrupt.

33. The system of claim 27 , wherein opcodes that are invalid in the context of the VM activate the software debugger processing logic.

34. The system of claim 27 , wherein the control logic includes at least one of a hardware control logic and a software control logic.

35. The system of claim 27 , further comprising an exception handler that bypasses the potentially unsafe instructions.

36. The system of claim 35 , wherein the exception handler is executed in the VM context.

37. The system of claim 35 , wherein the exception handler is executed in the VMM context.

38. A computing system for running a virtual machine, comprising:

a virtual machine monitor (VMM) that uses a software debugger;

a virtual machine (VM) that directly executes safe instructions; and

control logic that switches, at runtime to the software debugger for handling potentially unsafe instructions, and switches to direct execution to natively execute the safe instructions,

wherein the control logic (i) analyzes a source page of guest code to identify the potentially unsafe instructions;

(ii) generates a target page of the quest code with breakpoints replacing the potentially unsafe instructions;

(iii) executes instructions of the target page in a context of the VM;

(iv) groups the target pages in a frame;

(v) executes intra-frame transfers of control in the context of the VM; and

(vi) activates the software debugger to handle out-of-frame transfers of control.

39. A computer program product for running a virtual machine in a computing system, the computer program product comprising a computer useable medium having computer program logic recorded thereon for controlling at least one processor, the computer program logic comprising:

computer program code means for launching a virtual machine monitor (VMM) that uses a software debugger;

computer program code means for launching a virtual machine (VM) that can natively run safe instructions;

computer program code means for determining, at runtime, if the instruction is safe or potentially unsafe and (i) adjusting a context of the VM to enter an exception state during attempts to execute the potentially unsafe instructions, (ii) analyzing at runtime and while running the control logic, a context of the VM and a set of previously executed instructions, and (iii) correcting a state of the VM if the state of the VM was damaged by a potentially unsafe instruction from a set of previously executed instructions;

computer program code means for executing the safe instructions in a native mode of the VM; and

computer program code means for activating control logic to process potentially unsafe instructions in the software debugger.

40. A method for running a virtual machine in a computing system, comprising:

analyzing, at runtime, a source page of guest code to identify potentially unsafe instructions;

generating a target page of the guest code with breakpoints replacing the potentially unsafe instructions;

executing instructions of the target page in a context of the VM;

transferring control from a virtual machine (VM) running the guest code to a software debugger of a virtual machine monitor (VMM) when an exception is raised at one of the breakpoints;

grouping the target pages in a frame;

executing intra-frame transfers of control in the context of the VM;

activating the software debugger to handle out-of-frame transfers of control; and

bypassing at least one potentially unsafe instruction.

41. A computing system for running a virtual machine, comprising:

a virtual machine monitor (VMM) that uses a software debugger;

a virtual machine (VM) that executes safe instructions in a native mode; and

control logic that determines, at runtime, if the instruction is safe or potentially unsafe, and activates the software debugger in a VMM context to handle the potentially unsafe instructions,

wherein the control logic

(i) analyzes a source page of guest code to identify the potentially unsafe instructions;

(ii) generates a target page of the guest code with breakpoints replacing the potentially unsafe instructions;

(iii) executes instructions of the target page in a context of the VM;

(iv) groups the target pages in a frame;

(v) executes intra-frame transfers of control in the context of the VM; and

(vi) activates the software debugger to handle out-of-frame transfers of control, and

wherein the software debugger bypasses at least one of the potentially unsafe instructions.

42. A computing system for running a virtual machine, comprising:

a virtual machine monitor (VMM) that uses a software debugger;

a virtual machine (VM) that directly executes safe instructions; and

control logic that switches, at runtime to the software debugger for handling potentially unsafe instructions, and switches to direct execution to natively execute the safe instructions,

wherein the control logic

(i) analyzes a source page of guest code to identify the potentially unsafe instructions;

(ii) generates a target page of the guest code with breakpoints replacing the potentially unsafe instructions;

(iii) executes instructions of the target page in a context of the VM;

(iv) groups the target pages in a frame;

(v) executes intra-frame transfers of control in the context of the VM; and

(vi) activates the software debugger to handle out-of-frame transfers of control.

43. A computer program product for running a virtual machine in a computing system, the computer program product comprising a computer useable medium having computer program logic recorded thereon for controlling at least one processor, the computer program logic comprising:

computer program code means for launching a virtual machine monitor (VMM) that uses a software debugger;

computer program code means for launching a virtual machine (VM) that can natively run safe instructions;

computer program code means for determining, at runtime, if the instruction is safe or potentially unsafe;

computer program code means for executing the safe instructions in a native mode of the VM; and

computer program code means for activating control logic to process potentially unsafe instructions in the software debugger,

wherein the control logic

(i) analyzes a source page of guest code to identify the potentially unsafe instructions;

(ii) generates a target page of the guest code with breakpoints replacing the potentially unsafe instructions;

(iii) executes instructions of the target page in a context of the VM;

(iv) groups the target pages in a frame;

(v) executes intra-frame transfers of control in the context of the VM; and

(vi) activates the software debugger to handle out-of-frame transfers of control.

Assignments (16)