Description of feature
The create-gem.py script currently uses the pandas merge function, which is able to merge on any column. It turns out that the pandas join function is substantially faster, but is slightly more restrictive in that you can only join on the index.
join is essentially instantaneous, even for large matrices, where as merge becomes slow with large matrices. This is most important when you are dealing with massive matrices (greater than ~30,000 samples) where merge gets rather slow.
Description of feature
The create-gem.py script currently uses the
pandasmergefunction, which is able to merge on any column. It turns out that thepandasjoinfunction is substantially faster, but is slightly more restrictive in that you can only join on the index.joinis essentially instantaneous, even for large matrices, where asmergebecomes slow with large matrices. This is most important when you are dealing with massive matrices (greater than ~30,000 samples) where merge gets rather slow.