Merging data from a source location into a target location

1. A method for merging data from a source location into a target location containing existing data, wherein both the source location and the target location contain tuples of data, the method comprising:

for each tuple in the source location, during a single operation,

determining, by a processor, whether there is a matched tuple in the target location that satisfies a predetermined condition;

for each matched tuple that satisfies the predetermined condition, updating the matched tuple in the target location with a count value that is equal to a count of the matched tuple in the source location and the target location, wherein the updating comprises:

inserting the matched tuple into a delete vector;

marking the matched tuple in the delete vector as removed from the target location, without physically removing the matched tuple from the target location;

inserting the updated matched tuple into the target location; and

for each tuple that does not have a matched tuple that satisfies the predetermined condition, inserting the unmatched tuple into the target location;

associating a first indicator with the matched tuple in the target location, the first indicator indicating that the matched tuple in the target location is an existing tuple; and

associating a second indicator with the unmatched tuple inserted into the target location, the second indicator indicating that the unmatched tuple is new in the target location.

2. The method according to claim 1 , further comprising:

scanning both the source location and the target location to determine the tuples contained in each of the source location and the target location;

wherein determining whether there is a matched tuple in the target location that satisfies the predetermined condition comprises right outer joining the source location and the target location to determine which tuples from the source location are to be inserted into the target location and which tuples in the target location are to be updated.

3. The method according to claim 2 , wherein right outer joining the source location and the target location comprises adding a matching Boolean (M) to each output tuple of the right outer join to mark whether the tuple is new (M=false) or existing (M=true), wherein the first indicator is M=true, and the second indicator is M=false;

moving matched tuples (M=true) into the delete vector; and

inserting unmatched tuples (M=false) and matched tuples (M=true) into the target location.

4. The method according to claim 3 , further comprising:

applying a first filter to output the unmatched tuples into the delete vector;

applying expressions for the matched and unmatched tuples; and

applying a second filter to output the unmatched tuples and the matched tuples prior to inserting the unmatched tuples and the matched tuples into the target location.

5. The method according to claim 4 , further comprising:

implementing a sideways information passing operation that eliminates unmatched tuples from the target location before performing the right outer joining; and

wherein right outer joining the source location and the target location further comprises right outer joining the source location and the target location following implementation of the sideways information passing operation.

6. The method according to claim 1 , wherein the target location comprises a projection, the method further comprising:

scanning both the source location and the projection to determine the tuples contained in each of the source location and the projection;

wherein determining whether there is a matched tuple in the target location that satisfies the predetermined condition comprises determining whether there is a matched tuple in the projection by right outer merge joining the source location and the projection to determine which tuples from the source location are to be inserted into the projection and which tuples in the projection are to be updated.

7. The method according to claim 6 , wherein the target location comprises a plurality of projections, the method further comprising:

selecting one of the plurality of projections to scan based upon at least one factor, wherein the scanning comprises scanning both the source location and the selected one of the plurality of projections.

8. The method according to claim 7 , further comprising:

scanning a dimension table containing descriptive attributes of events;

joining the descriptive attributes of events in the dimension table with data contained in an additional projection with the right outer merge joined source location and the selected one of the plurality of projections;

inserting the unmatched tuples from the right outer merge join into each of the plurality of projections; and

updating the matched tuples from the right outer merge join in each of the plurality of projections.

9. The method according to claim 8 , wherein the existing data of the target location is distributed across different nodes, the method further comprising:

determining that the distributed existing data is to be redistributed among the different nodes; and

redistributing the distributed existing data among the different nodes in response to a determination that the distributed existing data is to be redistributed among the different nodes.

10. An apparatus for merging data from a source location into a target location containing existing data, wherein both the source location and the target location contain tuples of data, and wherein the target location comprises a column-based database, the apparatus comprising:

a processor;

a non-transitory storage medium storing instructions executable on the processor to:

scan the source location and the target location to determine the tuples contained in each of the source location and the target location;

right outer join the source location and the target location to determine unmatched tuples from the source location to be inserted into the target location, and matched tuples in the target location to be updated;

for each of the matched tuples in the target location that are to be updated, update the matched tuple in the target location with a count value that is equal to a number of times the matched tuple appears in the source location and in the target location, the updating comprising:

inserting the matched tuple into a delete vector;

marking the matched tuple in the delete vector as removed from the target location, without physically removing the matched tuple from the target location:

inserting the updated matched tuple into the target location; and

associating a first indicator with the updated matched tuple in the target location, the first indicator indicating that the matched tuple in the target location is an existing tuple; and

for each of the unmatched tuples in the source location that are to be inserted into the target location,

insert the unmatched tuple into the target location; and

associate a second indicator with the unmatched tuple inserted into the target location, the second indicator indicating that the unmatched tuple is new in the target location.

11. The apparatus according to claim 10 , wherein the instructions are executable on the processor to:

add a matching Boolean (M) to each output tuple of the right outer join to mark whether the output tuple is new (M=false) or existing (M=true), wherein the first indicator is M=true, and the second indicator is M=false;

insert existing tuples (M=true) into the delete vector; and

insert new tuples (M=false) and existing tuples (M=true) into the target location.

12. The apparatus according to claim 11 , wherein the instructions are executable on the processor to:

apply a first filter to output the new tuples into the delete vector;

apply expressions for the existing and new tuples; and

apply a second filter to output the new tuples and the existing tuples prior to inserting the new tuples and the existing tuples into the target location.

13. The apparatus according to claim 12 , wherein the instructions are executable on the processor to:

implement a sideways information passing operation that eliminates new tuples from the target location before performance of the right outer join.

14. The apparatus according to claim 10 , wherein the target location comprises at least one projection.

15. A non-transitory computer readable storage medium on which is stored machine readable instructions that when executed cause a system comprising a processor to merge data from a source location into a target location containing existing data, wherein both the source location and the target location contain tuples of data, and wherein the target location comprises a column-based database, the machine readable instructions when executed causing the system to:

scan the source location and the target location to determine the tuples contained in each of the source location and the target location;

right outer join the source location and the target location to determine unmatched tuples from the source location are to be inserted into the target location, and matched tuples in the target location are to be updated;

for each of the matched tuples in the target location that are to be updated, update the matched tuple in the target location with a count value that is equal to a sum of: a count of the matched tuple in the source location and a count of the matched tuple in the target location, the updating comprising:

inserting the matched tuple into a delete vector;

marking the matched tuple in the delete vector as removed from the target location, without physically removing the matched tuple from the target location;

inserting the updated matched tuple into the target location; and

associating a first indicator with the updated matched tuple in the target location, the first indicator indicating that the matched tuple in the target location is an existing tuple; and

for each of the unmatched tuples in the source location that are to be inserted into the target location,

insert the unmatched tuple into the target location; and

associate a second indicator with the unmatched tuple inserted into the target location, the second indicator indicating that the unmatched tuple is new in the target location.